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SELF-CONTAINED INTERACTIVE VIDEO DISPLAY SYSTEM 

CROSS-REFERENCE TO RELATED APPLICATION 
[0001] This application is a Continuation-in-Part Patent Application claiming 
5 priority from co-pending U.S. Patent Application No. 10/160,217, filed on May 28, 
2002, entitled "INTERACTIVE VIDEO DISPLAY SYSTEM," by Bell, and 
assigned to the assignee of the present application, which is herein incorporated 
by reference. This application also claims priority from co-pending U.S. 
Provisional Patent Application No. 60/504,375, filed on September 18, 2003, 

10 entitled "SELF-CONTAINED INTERACTIVE VIDEO DISPLAY SYSTEM," by Bell, 
and assigned to the assignee of the present application, from co-pending U.S. 
Provisional Patent Application No. 60/514,024, filed on October 24, 2003, 
entitled "METHOD AND SYSTEM FOR PROCESSING CAPTURED IMAGE 
INFORMATION IN AN INTERACTIVE VIDEO SYSTEM," by Bell, and assigned 

15 to the assignee of the present application, from co-pending U.S. Provisional 
Patent Application No. 60/528,439, filed on December 9, 2003, entitled "SELF- 
CONTAINED INTERACTIVE VIDEO DISPLAY SYSTEM AND FEATURES 
RELATING THERETO," by Bell, and assigned to the assignee of the present 
application, and from co-pending U.S. Provisional Patent Application No. 

20 60/554,520, filed on March 18, 2004, entitled "METHOD AND SYSTEM FOR 
ALLOWING A CAMERA TO VIEW AN AREA IN FRONT OF A DISPLAY BY 
IMAGING IT THROUGH THE DISPLAY," by Bell et al„ and assigned to the 
assignee of the present application, all of which are herein incorporated by 
reference. 
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FIELD 

[0002] The present writing relates to the field of visual electronic displays. 
Specifically, embodiments of the here described relate to a self-contained 
interactive video display system. 

BACKGROUND 

[0003] For many years, information was typically conveyed to an audience by 
use of static displays. For example, product advertisements were presented 
using print ads and posters. With the advent of television and movies, 
information could be presented using a dynamic display (e.g., commercials). 
While more engaging than static displays, dynamic displays do not typically 
provide interactivity between a user and the display. 

[0004] More recently, interactive touchscreens have been used for presenting 
information on flat surfaces. For example, an image may be displayed on a 
touchscreen, and a user may interact with the image by touching the 
touchscreen, causing the image to change. However, in order to interact with 
the image displayed on the touchscreen, the user must actually come in contact 
with the touchscreen. Moreover, typically touchscreens can only receive one 
input at any time, and are not able to discern the shape of the input. Essentially, 
current touchscreens are only able to receive the input of one finger contact. 
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[0005] In some applications, such as point-of-sale, retail advertising, promotions, 
arcade entertainment sites, etc., it is desirable to provide an interactive interface 
for displaying information to a user. This interactivity provides an even more 
engaging interface for presenting information (e.g., media, advertisements, etc.). 
5 By catching the attention of a person, for even a few moments, the person may 
be more likely to absorb the information presented in the interactive display than 
in previous displays. 



[0006] As described above, current interactive displays typically require a user to 
10 physically contact a touchscreen surface. By requiring contact with a 

touchscreen to provide interactivity, a large number of potential users are not 
interested in or intimidated by current interactive displays. Moreover, since only 
one user may interact with a touchscreen at a time, more users are excluded. 
Furthermore, because current touchscreens cannot discern the shape of input, 
1 5 they are limited in the type of information that can be presented in response to 
interaction. 



3 
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SUMMARY 

[0007] Various embodiments of the present invention, a self-contained 
interactive video display system, are described herein. A flat-panel display 
screen displays a visual image for presentation to a user on a front side of the 
5 flat-panel display screen. In one embodiment, the flat-panel display screen is a 
liquid crystal display (LCD) panel. A first illuminator illuminates the flat-panel 

4-. 

display screen with visible light. In one embodiment, the self-contained 
interactive video display system further comprises a diffusing screen for reducing 
glare of the first illuminator and illuminate the flat-panel display screen more 

10 evenly. A second illuminator illuminates an object. In one embodiment, the 
object is a body part of a human user. In one embodiment, the second 
illuminator projects illumination through the flat-panel display screen onto the 
object. In one embodiment, the second illuminator is positioned so as to reduce 
potential for glare effects on the camera. In one embodiment, the second 

1 5 illuminator is located next to the flat-panel display screen such that the second 
illuminator does not project illumination through the flat-panel display screen. In 
one embodiment, the self-contained interactive video display system comprises 
a plurality of the second illuminators, wherein the second illuminators are located 
next to the flat-panel display screen as well as behind the screen. In one 

20 embodiment, the second illuminator is strobed in time with exposures of the 
camera. 



[0008] A rcamera detects interaction of an illuminated object with the visual 
image, wherein the camera is operable to view the object through the flat-panel 

4 
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display screen. In one embodiment, the second illuminator is an infrared 
illuminator for illuminating the object with infrared illumination, and wherein the 
camera is an infrared camera for detecting infrared illumination. In one 
embodiment, the camera is located behind the flat-panel display screen and 
5 pointed toward the screen, allowing the camera to view the area on and in front 
of the screen. In one embodiment, the camera's image is calibrated to the visual 
image such that the interaction caused by the object is matched to a physical 
position of the object proximate to the screen. In one embodiment, the camera 
and the second illuminator comprise a time-of-flight camera. In one 
10 embodiment, a plurality of time-of-flight cameras are placed in a manner so as to 
provide complete coverage of the area in front of the display and angled so as to 
prevent specular reflection into the time-of-flight cameras due to the screen. 

[0009] A computer system directs the projector to change the visual image in 
1 5 response to the interaction. In one embodiment, the camera, the first illuminator, 
the second illuminator, and the computer system are comprised within an 
enclosure, and wherein one side of the enclosure comprises the flat-panel 
display screen. 

20 [0010] In one embodiment, the self-contained interactive video display system 
further comprises a series of mirror strips positioned at a distance from the 
screen to correct distortion of the camera's view. In another embodiment, the 
self-contained interactive video display system further comprises a Fresnel lens 
positioned adjacent to the screen to correct distortion of a view of the camera. In 

5 
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one embodiment, the self-contained interactive video display system further 
comprises a wavelength-based diffuser positioned adjacent to the flat-panel 
display screen. In one embodiment, the diffuser is substantially transparent to 
infrared light and substantially translucent to visible light. In another 
5 embodiment, the diffuser is a material with Rayleigh scattering. In one 
embodiment, the self-contained interactive video display system further 
comprises a diffuser having a physical texture positioned adjacent to the flat- 
panel display screen, wherein the diffuser is substantially translucent to light 
passing through the diffuser at an oblique angle and substantially transparent to 
10 light passing through the diffuser at a substantially perpendicular angle, wherein 
the first illuminator is placed at an oblique angle to the diffuser. 



[001 1] In one embodiment, the self-contained interactive video display system 
further comprises a scattering polarizer positioned adjacent to the flat-panel 

15 display screen and for scattering light from the first illuminator, and wherein the 
camera is sensitive to light of a polarization not scattered by the scattering 
polarizer. In one embodiment, where the flat-panel display is a liquid crystal 
display panel, the scattering polarizer is oriented such that light polarized in a 
direction for which the scattering polarizer scatters light passes through the liquid 

20 crystal display panel and light polarized in a direction for which the scattering 

polarizer does not scatter light is absorbed by the liquid crystal display panel.. In 
another embodiment, the self-contained interactive video display system further 
comprises a linear polarizer for polarizing light received at the camera at 
wavelengths to which the camera is sensitive, so as to allow the camera to 
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ignore light scattered by the scattering polarizer. In one embodiment, the self- 
contained interactive video display system further comprises a diffusing material 
that can change from substantially translucent to substantially transparent, is 
substantially translucent when the first illuminator is illuminating the display, and 
5 is substantially transparent when the camera is detecting objects in front of the 
flat-panel display screen, wherein the diffusing material is placed behind the flat- 
panel display screen. 

[0012] In one embodiment, the self-contained interactive video display system is 
10 operable to determine information about the distance of the object from the 
screen. In one embodiment, the camera is a stereo camera. In another 
embodiment, the camera is a time-of-flight camera. In one embodiment, the 
time-of-flight camera is positioned such that the time-of-flight camera does not 
reflect back onto itself. 

15 

[0013] In one embodiment, the self-contained interactive video display system 
provides touchscreen functionality when the object is touching the screen. In 
one embodiment, the self-contained interactive video display system further 
comprises a transparent touchscreen adjacent the front side of the screen. In 
20 another embodiment, the self-contained interactive video display system further 
comprises an edge-lit transparent sheet adjacent the front side of the screen, 
and wherein the camera is operable to distinguish light created when the object 
comes in contact with the edge-lit transparent sheet. 
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[0014] In another embodiment, the present invention provides a method for 
presenting an interactive visual image using a self-contained interactive video 
display system. A visual image is displayed on a flat-panel display screen for 
presentation to a user on a front side of the flat-panel display screen. A back 
5 side of the flat-panel display screen is illuminated with visible light. An object 
proximate the front side of the flat-panel display screen is illuminated from a 
second illumination source. Interaction of the object with the visual image is 
detected by a device able to sense its presence through the flat-panel display 
screen. The visual image is changed in response to the interaction. 

10 
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BRIEF DESCRIPTION OF THE DRAWINGS 

[0015] The accompanying drawings, which are incorporated in and form a part 
of this specification, illustrate embodiments of the invention and, together with 
the description, serve to explain the principles of the invention: 

5 

[0016] Figure 1 shows one physical configuration of the components of an 
interactive video system, in accordance with an embodiment of the present 
invention. 

10 [0017] Figure 2 shows one arrangement of a screen where linear polarizer 
sheets are used to eliminate or reduce glare, in accordance with one 
embodiment of the present invention. 

[0018] Figure 3 shows cross sections of several other configurations of the 
15 interactive video system, in accordance with various embodiments of the present 
invention. 

[0019] Figures 4A and 4B are schematic diagrams respectively illustrating 
embodiments of an interactive video system, in accordance with embodiments of 
20 the present invention. 

[0020] Figures 5A and 5B are schematic diagrams respectively illustrating 
embodiments of an interactive video system, in accordance with embodiments of 
the present invention. 
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[0021] Figures 6A and 6B are schematic diagrams respectively illustrating two 
configurations of off-axis projection, in accordance with embodiments of the 
present invention. 

5 

[0022] Figures 7 A and 7B are schematic diagrams illustrating an interactive flat- 
panel display system, in accordance with one embodiment of the present 
invention. 

10 [0023] Figure 8A is a schematic diagram illustrating a technique for reducing 

image distortion using a Fresnel lens, in accordance with one embodiment of the 
present invention. 

[0024] Figure 8B is a schematic diagram illustrating a technique for reducing 
15 image distortion using a series of mirror strips, in accordance with one 
embodiment of the present invention. 

[0025] Figures 9A and 9B illustrate a schematic layout of an interactive video 
display system having a scattering polarizer screen, in accordance with an 
20 embodiment of the present invention. 

[0026] Figure 10A illustrate a cross-section of a screen with microscopic 
scattering ridges or bumps, in accordance with an embodiment of the present 
invention. 

10 
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[0027] Figure 10B illustrate a cross-section of a screen with microscopic 
scattering pits or grooves, in accordance with an embodiment of the present 
invention. 

5 

[0028] Figure 1 1 illustrates a sample configuration for edge lighting, in 
accordance with an embodiment of the present invention. 

[0029] Figure 12A illustrates a flat-panel display cross-section, in accordance 
10 with an embodiment of the present invention. 

[0030] Figure 12B illustrates a flat-panel display cross-section, in accordance 
with another embodiment of the present invention. 

15 [0031] Figure 13 illustrates a camera and illumination subsystem, in accordance 
with an embodiment of the present invention. 

[0032] Figure 14 illustrates an illumination subsystem for camera utilizing tilted 
scattering polarizer, in accordance with an embodiment of the present invention. 

20 

[0033] Figure 15 illustrates a camera and illumination subsystem for time-of- 
flight cameras, in accordance with an embodiment of the present invention. 



11 



WO 2005/057399 



PCT/US2004/041320 



[0034] Figure 16 shows a first configuration for capturing 3D data, in accordance 
with an embodiment of the present invention. 

[0035] Figure 17 shows a second configuration for capturing 3D data, in 
5 accordance with an embodiment of the present invention. 

[0036] Figure 18A shows two additional configurations for capturing 3D data, in 
accordance with an embodiment of the present invention. 

10 [0037] Figure 18B shows another configuration for capturing 3D data, in 
accordance with an embodiment of the present invention. 

[0038] Figures 19A and 19B are schematic diagrams illustrating light scattering, 
in accordance with an embodiment of the present invention. 

15 

[0039] Figure 20A illustrates high distortion, in accordance with an embodiment 
of the present invention. 

[0040] Figure 20B illustrates reduced distortion by distancing camera from 
20 display screen, in accordance with an embodiment of the present invention. 

[0041] Figure 21 A illustrates distortion reduction using Fresnel lens, in 
accordance with an embodiment of the present invention. 
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[0042] Figure 21 B illustrates distortion elimination using Fresnel lens, in 
accordance with an embodiment of the present invention. 

[0043] Figure 21 C shows the use of Fresnel lenses to eliminate distortion in a 
5 two-camera system, in accordance with an embodiment of the present invention. 

[0044] Figure 22 is a schematic diagram illustrating a window display, in 
accordance with one embodiment of the present invention. 

10 [0045] Figures 23A, 23B, and 23C are schematic diagrams respectively 

illustrating various techniques for reducing glare, in accordance with different 
embodiments of the present invention. 

[0046] Figures 24A and 24B are schematic diagrams illustrating a technique for 
15 reducing glare using view control film, in accordance with embodiments of the 
present invention. 

[0047] Figure 25 illustrates a cross-section of one configuration of window 
display using scattering polarizer, in accordance with an embodiment of the 
20 present invention. 

[0048] Figure 26 illustrates a cross-section of one configuration of window 
display using scattering polarizer and a micro-prism material, in accordance with 
an embodiment of the present invention. 
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[0049] Figure 27 illustrates a cross-section of one configuration of window 
display using a mirror for compaction purposes, in accordance with an 
embodiment of the present invention. 

5 

[0050] Figure 28 illustrates a side view of an interactive display including 
multiple time-of-flight cameras, in accordance with an embodiment of the present 
invention. 

10 [0051] Figure 29 illustrates a top view of an interactive display including multiple 
time-of-flight cameras, in accordance with an embodiment of the present 
invention. 
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DETAILED DESCRIPTION 

[0052] Reference will now be made in detail to various embodiments of the 
invention, an electronic device for monitoring the presence of objects around a 
second electronic device, examples of which are illustrated in the accompanying 
5 drawings. While the invention will be described in conjunction with these 

embodiments, it is understood that they are not intended to limit the invention to 
these embodiments. On the contrary, the invention is intended to cover 
alternatives, modifications and equivalents, which may be included within the 
spirit and scope of the invention as defined by the appended claims. 

10 Furthermore, in the following detailed description of the invention, numerous 
specific details are set forth in order to provide a thorough understanding of the 
invention. However, it will be recognized by one of ordinary skill in the art that 
the invention may be practiced without these specific details. In other instances, 
well known methods, procedures, components, and circuits have not been 

15 described in detail as not to unnecessarily obscure aspects of the invention. 

[0053] Some portions of the detailed descriptions, which follow, are presented in 
terms of procedures, steps, logic blocks, processing, and other symbolic 
representations of operations on data bits that can be performed on computer 
20 memory. These descriptions and representations are the means used by those 
skilled in the data processing arts to most effectively convey the substance of 
their work to others skilled in the art. A procedure, computer executed step, logic 
block, process, etc., is here, and generally, conceived to be a self-consistent 
sequence of steps or instructions leading to a desired result. The steps are 

15 
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those requiring physical manipulations of physical quantities. Usually, though not 
necessarily, these quantities take the form of electrical or magnetic signals 
capable of being stored, transferred, combined, compared, and otherwise 
manipulated in a computer system. It has proven convenient at times, principally 
5 for reasons of common usage, to refer to these signals as bits, values, elements, 
symbols, characters, terms, numbers, or the like. 

[0054] It should be borne in mind, however, that all of these and similar terms 
are to be associated with the appropriate physical quantities and are merely 

10 convenient labels applied to these quantities. Unless specifically stated 
otherwise as apparent from the following discussions, it is appreciated that 
throughout the present invention, discussions utilizing terms such as "projecting" 
or "detecting" or "changing" or "illuminating" or "correcting" or "eliminating" or the 
like, refer to the action and processes of an electronic system (e.g., interactive 

15 video system 100 of Figure 1), or similar electronic computing device, that 

manipulates and transforms data represented as physical (electronic) quantities 
within the electronic device's registers and memories into other data similarly 
represented as physical quantities within the electronic device memories or 
registers or other such information storage, transmission or display devices. 

20 

[0055] Various embodiments of the present invention, a self-contained 
interactive video display system, are described herein. In one embodiment, a 
flat-panel display screen displays a visual image for presentation to a user on a 
front side of the flat-panel display screen. A first illuminator illuminates the flat- 

16 
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panel display screen with visible light. A second illuminator illuminates an object. 
A camera detects interaction of an illuminated object with the visual image, 
wherein the camera is operable to view the object through the flat-panel display 
screen. A computer system directs the projector to change the visual image in 
5 response to the interaction. 

Interactive Video Projection System 

[0056] The present invention in the form of one or more exemplary 
embodiments will now be described. According to one exemplary embodiment, 

10 an interactive video system 100 as shown in Figure 1 is provided. The 

interactive video system 100 uses an camera 115 fitted with a filter that blocks 
visible light, an illuminator 125 that illuminates screen 130 being viewed by 
camera 1 15, a projector 120 that projects an image onto the interactive space of 
screen 1 30, and a computer 110 that takes as input the image of camera* 115 

15 and outputs a video image to projector 120. In one embodiment, illuminator 125 
is an infrared illuminator and camera 115 is an infrared camera operable to 
record images illuminated by the infrared light of illuminator 125. It should be 
appreciated that camera 115 and illuminator 125 can be configured to operate 
using any form of light that is not visible, and is not limited to infrared light. 

20 

[0057] Computer 110 processes the camera 115 input to discern on a pixel-by- 
pixel basis what portions of the volume in front of screen 130 are occupied by 
people (or moving objects) and what portions of screen 130 are background. 
Computer 110 accomplishes this by developing several evolving models of what 

17 



WO 2005/057399 



PCT/US2004/041320 



the background is supposed to look like, and then comparing its concepts of the 
background to what camera 115 is currently seeing. Alternatively, components 
of computer 110 that process camera 115 input are collectively known as the 
vision system. Various embodiments of this vision system are described in co- 
5 pending U.S. Patent Application No. 10/160,217, filed on May 28, 2002, entitled 
"INTERACTIVE VIDEO DISPLAY SYSTEM," by Bell, and assigned to the 
assignee of the present application, in co-pending U.S. Provisional Patent 
Application No. 60/504,375, filed on September 18, 2003, entitled "SELF- 
CONTAINED INTERACTIVE DISPLAY SYSTEM," by Bell, and assigned to the 
10 assignee of the present application, and in co-pending U.S. Provisional Patent 
Application No. 60/514,024, filed on October 24, 2003, entitled "METHOD AND 
SYSTEM FOR PROCESSING CAPTURED IMAGE INFORMATION IN AN 
INTERACTIVE VIDEO SYSTEM," by Bell, and assigned to the assignee of the 
present application, all of which are herein incorporated by reference. 

15 

[0058] The evolving background is an important part of the vision, as it allows 
the system to be resilient to changes in lighting, scuff marks on the screen, and 
other disturbances. The output of the vision system is a black and white mask 
image that feeds into an effects engine, which also runs on computer 110. The 
20 effects engine runs the applications that create the interactive graphics on 

screen 130. Artists can design effects using a large variety of effect components 
as well as scripting, allowing them to create a wide variety of interactive 
experiences. Finally, the images created by the effects engine are outputted to 
projector 120. 
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[0059] It is desirable that all the electronic components of interactive video 
system 100 (e.g., camera 115, projector 120, computer 110, and illuminator 125) 
are on one side of screen 130 while the user interaction takes place on the other 
5 side of screen 130. In one embodiment, screen 130 is partially translucent for 
the light of projector 120 to allow an image to form on the surface of screen 130. 
However, screen 130 is also partially transparent to camera 1 15 so that camera 
115 can see objects on the opposite side of screen 130. It should be 
appreciated that the terms transparent and translucent as referred to throughout 

10 the current specification are defined as meaning at least partially transparent 
and/or translucent, respectively. It should also be appreciated that the terms 
"scattered" and "not scattered" as referred to throughout the current specification 
are defined as meaning "substantially scattered" and "not substantially scattered" 
respectively. Finally, it should also be appreciated that the terms "diffused" and 

1 5 "not diffused" as referred to throughout the current specification are defined as 
meaning "substantially diffused" and "not substantially diffused" respectively. 

[0060] Figure 1 shows one physical configuration of the components of an 
exemplary embodiment of the present invention. All sensing and display 
20 components, including camera 115, illuminator 125, computer 110, and projector 
120, are inside a box 140. In one embodiment, all sides of the box 140 are 
opaque except for one side. This one side which is not opaque is a screen 130 
for displaying the projected image. 



19 



WO 2005/057399 PCT/US2004/041320 



[0061] In one embodiment, smooth, flat materials that have strong Rayleigh 
scattering and relatively little scattering of other forms are used for screen 130. If 
light is scattered by screen 130 (a translucent screen), then that light will be 
visible as an image on screen 130. If light is not scattered or absorbed by 
5 screen 130 (a transparent screen), then the light will pass straight through 
screen 130 like a pane of glass. 

[0062] Rayleigh scattering is proportional to 1/(wavelength A 4), which means that 
light with short wavelengths is scattered much more than light with long 
wavelengths. Thus, infrared light, which has a wavelength greater than 800 
nanometers (nm), is scattered much less than visible light, which has a 
wavelength of 400 nm - 700 nm. In the present embodiment, projector 120 uses 
visible light, while camera 115 uses infrared light, allowing camera 115 to see 
through screen 130 while light emitted by projector 120 is scattered onto screen 
130. In one embodiment, the material of screen 130 is smooth and homogenous 
down to a scale of preferably around 40nm, in order to have good Rayleigh 
scattering but minimal scattering of other kinds 

[0063] In one embodiment, the screen material has a fine-scale structure that 
20 causes most visible light to scatter. However, it also is not be too dense or thick; 
otherwise, most of the infrared light will scatter as well. In addition, the material 
should not absorb much visible or infrared light; otherwise, this will make the 
material opaque and therefore a poor screen. One example of a material that 
satisfies the property of strong Rayleigh scattering is an ordinary white plastic 

20 
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trash bag. In one embodiment, screen 130 is created by sandwiching the bag 
between two sheets of glass. Another example of a material that satisfies this 
property is polyethylene sheeting. 

5 [0064] Increasing the wavelength of illuminator 125 and camera 1 1 5's filter 
improves the performance of interactive video system 100 because (with the 
appropriate screen material and thickness chosen) the increased wavelength 
maximizes the amount of scattering of visible light (which minimizes glare) and 
minimizes the amount of scattering of infrared light (which improves camera 
10 1 15's view of objects above the screen). In one embodiment, a 950nm LED 
cluster illuminator and a monochrome Charged Coupled Device (CCD) camera 
with a 40nm width 950nm center bandpass filter at the front of its lens are used. 

[0065] Several features can be added to interactive video system 100 to further 
15 enhance its performance. 

Reducing Glare on the Camera 

[0066] There may be reflected glare from illuminator 125 onto camera 115. This 
glare can interfere with camera 1 15's ability to see beyond screen 130. In one 
20 embodiment, a near-infrared antireflective coating is placed on the bottom and/or 
top of screen 130 to mitigate this interference and improve the performance of 
camera 115. In addition, illuminators 125 can be placed at an oblique angle 
relative to screen 130, preventing any specular reflection from taking place. 

21 



WO 2005/057399 



PCT/US2004/041320 



[0067] Furthermore, in another embodiment, infrared linear polarizing filters are 
added to the front of illuminator 125 and camera 115 (with the orientation of the 
polarization of illuminator 125 perpendicular to the polarization of camera 1 15) to 
further reduce glare. This glare happens because light that reflects directly off 
5 the bottom of screen 130 will still be polarized, while light that hits an object 
outside screen 130 loses its polarization. 

Directional Ambient Infrared 

[0068] Ambient sources of infrared light can pose a problem for the vision 
10 system of interactive video system 100. For example, if a bright external infrared 
source is shining on the display from one direction, any object between this 
infrared source and the screen will cast an infrared shadow onto screen 130. 
The vision system may confuse this with an actual object on screen 1 30, causing 
the application to malfunction. Several techniques can be used to reduce the 
15 problem of infrared shadows. 



[0069] In one embodiment, the wavelength of the illuminator 125 can be chosen 
to be as uniform as possible. A narrow bandpass filter, which only passes light 
of the wavelengths put out most strongly by illuminator 125, can be added to the 
20 front of camera 115. 

[0070] In another embodiment, the use of a patterned illuminator allows the 
system to distinguish between infrared shadows and actual objects on screen 
130. For additional details, see U.S. Patent Application No. 10/160,217, filed 
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May 28, 2002, entitled "INTERACTIVE VIDEO DISPLAY SYSTEM", by Bell, 
which is herein incorporated by reference. 

[0071] In another embodiment, illuminator 125 and camera 115 can be strobed. 
5 Some illumination sources, such as light emitting diodes (LEDs), can turn on far 
brighter for brief periods than they can continuously. If illuminator 125 is turned 
on only during the exposure of camera 115, and the camera exposure is brief 
enough, the brightness of illuminator 125 is greatly magnified relative to the 
ambient light. This is true because the image of camera 1 15 during a short 
10 exposure in which illuminator 125 is turned on very brightly will contain much less 
ambient light but nearly as much light from illuminator 125 as compared to an 
image from a longer exposure in which illuminator 125 is on continuously but at 
a lower, continuous-duty brightness. 

15 [0072] Camera 115 and illuminator 125 can be synchronized. For example, a 
microcontroller or other electronic circuit can either read or set the camera 
exposure sync and trigger pulsed power to illuminator 125 at the appropriate 
time. 

20 [0073] The performance of strobing can be further improved by only turning on 
illuminator 125 during every second camera exposure. Thus, camera 115 would 
alternate between an exposure with illuminator 125 on and one with illuminator 
125 off. Since the goal is to remove the ambient infrared light, computer 110 can 
continuously generate an image with no ambient light by taking the difference 
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between the current image and the previous image. Because illuminator 1 25 is 
lit only on every second frame, one image will have only ambient infrared light 
while the other will have the ambient infrared light plus the light of illuminator 
125. By taking the pixel-wise difference between the current and previous 
5 images, the ambient infrared can be canceled out, leaving only the light of 
illuminator 125. 

[0074] In the case of an interlaced CCD camera, flashing illuminator 125 on 
during alternate exposures would produce camera output images in which the 

10 even-numbered lines have illuminator 125 on and the odd-numbered lines have 
illuminator 125 off. Thus, instead of comparing two images, computer 110 can 
take the difference between the odd numbered lines and the even numbered 
lines to subtract out the ambient light. The strobing could be performed using 
two cameras 115, timed so that the first and second cameras take their 

1 5 exposures at slightly different times, and the illuminator 125 is only on for one of 
the two exposures. Alternately, the two cameras could be sensitive to slightly 
different wavelengths, and the illuminator 125 only emits light at the second 
wavelength. 



20 [0075] In another embodiment, in an environment with no ambient infrared light, 
strobing illuminator 125 for only every second exposure reduces the system's 
reaction time. Any movement when illuminator 125 is off will not be noticed 
during the second exposure. However, this can be improved by turning only part 
of illuminator 125 off, or simply reducing the power pf illuminator 125, during 
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every second exposure. Then, illuminator 125 alternates between "all the way 
on" and "partly on". When computer 110 takes the difference between the 
current exposure and the previous exposure, the result will contain no ambient 
infrared and part of the light of illuminator 125. This configuration will provide the 
5 fastest possible reaction time for the user in both environments with no ambient 
infrared and some ambient infrared. 

Projector Glare 

[0076] Because the screen material is not completely translucent, some of the 
10 light from projector 120 may pass directly through screen 130. As a result, 
projector 120 may cause glare in the user's eye. In one embodiment, by 
making the wavelength of illuminator 125 longer and using a screen 130 that 
causes more scattering, camera 115 is still able to see through screen 130 while 
the amount of visible light glare is reduced. 

15 

[0077] In another embodiment, linear polarizer sheets can be used to eliminate 
or reduce the glare. Figure 2 shows one arrangement where linear polarizer 
sheets are used to eliminate or reduce the glare. A vertically polarized sheet 230 
and a horizontally polarized sheet 220 are placed immediately below and above 
20 screen 210, respectively. As the projected light passes through vertically 
polarized sheet 230, it becomes vertically polarized. Since scattering 
depolarizes the light, much of the scattered light on screen 210 is still visible to 
the viewer. However, the light not scattered by screen 210 (which causes the 
glare) is absorbed almost entirely by horizontally polarized sheet 220 because 
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the light is vertically polarized. Thus, the glare is eliminated while screen 210 
remains bright. Note that if the camera is sensitive to infrared light, a linear 
polarizing material can be chosen that does not polarize infrared light. 

5 [0078] In another embodiment, if the projector is a liquid crystal display (LCD) 
projector, the light will already be polarized. For some LCD projectors, red, 
green, and blue light are all polarized in the same direction. In this case, a 
polarizing film is not needed under the screen. In some LCD projectors, red and 
blue are polarized in one direction, while green is polarized 90 degrees off from 

10 that direction. In this case, in one embodiment, the polarized sheets are present 
and should be polarized to 45 and 135 degrees off from the red-blue direction. 
In another embodiment, a color selective polarization rotator can be placed on or 
inside the projector to get the red, green, and blue light polarized in the same 
direction. In this case, only one linear polarizer in front of the screen is needed. 

15 A color selective polarization rotator, such as the retarder stack "ColorSelect" 
technology produced by the ColorLink Corporation, is used to rotate the 
polarization of green light by 90 degrees. Alternatively, the polarization of red 
and blue light can be rotated by 90 degrees to achieve the same effect. 



20 Physical Configurations 

[0079] There are multiple potential physical configurations of the interactive 
video display system. One configuration is the tabletop display, as shown and 
described in Figure 1 . The interactive video display system sits on a surface, 
has all of the electronics contained inside of a box, is several feet tall, and has a . 
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horizontal screen on top of the box. However, the interactive video display 
system can also be used to create diagonal, vertical, or curved displays. 

[0080] A portion of the physical space taken up by the interactive video display 
5 system is simply dead space - in order to have a reasonably large image on the 
screen, the projector needs to be a significant distance away from the screen. 
This distance can be decreased through the use of mirrors; this allows the 
projector's beam to be redirected and fit into a more compact space. In one 
embodiment, the camera can be mounted at different points in the box and may 
10 view the screen through a mirror, so long as it has a clear view of the screen. In 
one embodiment, the infrared illuminator can be mounted anywhere in the box, 
or even on the surface of the box, so long as it illuminates objects above the box. 

[0081] Figure 3 shows cross sections of several other potential configurations of 
15 the system. As all parts can be easily secured, the designs shown can be 
rotated to any direction. Display 310 illustrates the interactive video display 
described at Figure 1. Displays 320 and 330 illustrate interactive video displays 
that are more compact using mirrors to redirect the projector's beam. Displays 
340 and 350 illustrate interactive video displays that are angled, using an angled 
20 camera (display 340) and using a mirror to redirect the projector's beam (display 
350). Display 360 illustrates an interactive video display using multiple mirrors to 
redirect the projector's beam. 
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Additional Configurations of an Interactive Video Display 
[0082] According to one aspect of the present invention, a number of exemplary 
methods for lighting an area in front of a screen are provided. In a self-contained 
interactive video display, an infrared camera, infrared illuminator, and a visible- 
5 light projector are all on one side of a screen while the user is on the other. In 
order to provide the desired functionality, the screen material that is used is 
mostly translucent (but may also be slightly transparent) to visible light, and is 
also preferably mostly transparent to infrared light (referred to hereinafter as "IR- 
Transparent VIS-Translucent screen" or "Main screen"). Light emitted by the 
10 infrared illuminator scatters to a certain degree when it passes through the 
screen. This light is picked up by the camera and may cause the camera's 
image to be low contrast and washed-out. As a result, the camera's view of the 
objects beyond the screen may be impeded, which results in reduced 
performance characteristics. 

15 

[0083] The present invention addresses the foregoing problem in a number of 
ways. In one embodiment, the infrared illuminator can be placed as close to the 
screen as possible. For example, the illuminators may be placed directly against 
the screen along the border. This configuration is shown in Figure 4A. The 
20 material in front of the illuminators (referred to as "cover for illuminator 402" in 
Figure 4A) may include any material that is at least somewhat translucent or 
transparent to infrared. Options for cover for illuminator 402 material include the 
main screen material, a clear transparent material, or a black opaque material 
that is transparent to infrared. References to cover for illuminator in subsequent 
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figures have similar meaning. A physical block may be used to prevent infrared 
light spillage onto the main screen. 

[0084] However, the above embodiment may result in poor illumination of 
5 objects that are close to the screen and near the center of the screen. This is 
because light shined on most materials at an oblique angle tends to reflect off of 
the material's surface rather than pass through, e.g., the materials transparency 
is functionally reduced. One way of addressing this is to simply move the screen 
back from the surface of the display, thus allowing the infrared illuminators to 
10 shine through the screen from a less oblique angle. This configuration is shown 
in Figure 4B. 

[0085] In another embodiment, the illuminators can be placed in front of the 
screen in a way that allows them to easily illuminate all locations in front of the 

1 5 screen. One configuration, in which the illuminator protrudes from the front of 
the screen, is shown in Figure 5A; another configuration, in which the display 
surface is recessed, is shown in Figure 5B.[0018] In the embodiments described 
in Figures 4A, 4B, 5A and 5B, the illuminators may be placed at regular intervals 
around the screen, in a continuous line, or at strategic locations. These 

20 illumination strategies shown in Figures 4A, 4B, 5A, and 5B can also be 

combined with illuminators that are behind the screen and shine through the 
screen. 
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Off-Axis Projection 

[0086] According to another aspect of the present invention, off-axis projection 
is used to improve the performance of the self-contained interactive video display 
system. Off-axis video projectors are capable of projecting a rectangular video 
5 image onto a flat surface at an oblique angle. These off-axis projectors are 
extremely important for interactive displays, as they allow the size of the overall 
system to shrink dramatically and they allow the glare to be reduced. 

[0087] Figures 6A and 6B show two configurations of a self-contained interactive 
10 video display using an off-axis projector. Glare is reduced when using an IR- 
transparent, VIS-translucent screen with an off-axis projector. Since the screen 
is not perfectly translucent to visible light (nor perfectly transparent to infrared 
light), some visible light will pass straight through the screen. If the screen is 
thickened, then more of the remaining visible light will be scattered, reducing the 
15 glare. Making the screen thicker also makes the screen less transparent to 

infrared since the screen is not perfectly transparent to infrared. However, if the 
visible light passes through the screen at an oblique angle instead of a 
perpendicular angle, then the light has to travel through the screen for a greater 
distance, reducing the amount of glare. For example, light passing through the 
20 screen at 30 degrees from parallel to the screen has to pass through twice as 
much screen material as light passing through at a perpendicular angle to the 
screen. Thus, if the infrared camera views the screen directly while the visible 
light projector shines light on the screen from an oblique angle, maximum 
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transparency for the infrared camera and maximum translucency for the visible- 
light projector can be obtained. 

Transparent Flat-Panel Displays 
5 [0088] The self-contained interactive video display can be implemented with 
display technologies other than video projectors. Any flat-panel display that is at 
least partially transparent to light visible to the camera may be implemented if it 
is used in place of the main screen. For example, the Transparent Imaging 
Matrix, a form of LCD panel sold by Provision, can be used as a display in an 

10 embodiment where the camera is a near-infrared camera. This form of LCD 

panel is clear and transparent when the color being displayed is white. It is also 
transparent in near-infrared, no matter what color is being displayed. Many 
types of LCD panels, including the transmissive LCD panels used in laptop 
monitors, flat-panel LCD computer monitors, and flat-panel LCD TV screens, 

1 5 also have the property that they are transparent in near-infrared. 

[0089] The flat-panel display that is at least partially transparent to light visible to 
the camera will be referenced in this text as the "transparent flat-panel display." 
Although the examples described herein involve transparent flat-panel displays 
20 that are completely transparent in infrared and an infrared camera, the 

implementation applies equally well to a camera that functions in a different 
wavelength range and a transparent flat-panel display that is completely 
transparent to light detectable by that camera. 
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[0090] Using a transparent LCD panel or other flat-panel display technology that 
is at least partially transparent to infrared, an interactive flat-panel display can be 
constructed using an infrared camera. Transparent LCDs typically lack their own 
illumination, so they may have to be illuminated by an external source. In one 
embodiment, this external source includes a white visible-light illuminator behind 
the LCD panel. In one embodiment, a screen that is transparent to the camera 
but scatters the visible-light illuminator is placed immediately behind the LCD 
panel so as to diffuse the light of the illuminator more easily. 

[0091] Figure 7A illustrates an exemplary transparent interactive flat-panel 
display system 700, in accordance with a an embodiment of the present 
invention. In one embodiment, the appearance of display 700 is improved by 
placing an IR-transparent VIS-translucent screen 720 material behind the 
transparent flat-panel display 710. Then, shining any light onto screen 720 will 
illuminate the display 710 in a more diffuse manner. In order to make the lighting 
of screen 720 maximally diffuse, system 700 may use lights 730 that shine onto 
the screen from an oblique angle or lights 730 that have separate diffusers 740 
in front of them. Note that diffusers 740 do not block the view of camera 760. 
Figure 7B shows the same configuration in a cutaway top view, with transparent 
flat-panel display 710 and screen 720 removed. The visible light illuminator 730 
may include any lighting technology capable of producing visible light, including 
LEDs, fluorescents, neon tubes, electroluminescent wire or sheeting, halogen 
lights, and incandescent bulbs. The infrared illuminator 750 may include any 
lighting technology capable of producing infrared light visible to the camera, 
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including LEDs, heat lamps, halogen lamps, or incandescent lamps. The 
infrared and visible light may both be produced by the same light source. 
Alternately, for greater control of the infrared light, a film that is transparent to 
visible light but opaque to infrared light may be placed over visible illuminators 
5 730. 

[0092] The improvements noted for the projector based system, including the 
strobing techniques mentioned in the section titled "Directional Ambient Infrared", 
the physical arrangements mentioned in the section titled "Physical 
10 Configurations", and the illuminator arrangements mentioned in the section titled 
"Additional Configurations of an Interactive Video Display", are all applicable to 
the transparent flat-panel display based systems described in this section. 

[0093] The use of a transparent flat-panel display rather than a projector allows 
1 5 the interactive video display to be significantly reduced in size. However, this 
poses a problem for the computer vision system, which must be able to see 
through the screen. If there is only a small distance between the screen and the 
back of the display box, then camera 760 would have to be extremely wide-angle 
in order to view the full area in front of screen 720, the location of objects that the 
20 system should detect, such as the hand of a user. This may pose issues 
because of the difficulties in looking through the screen at an oblique angle. 

[0094] One way of resolving the issue of illumination at an oblique angle is the 
use of polarizing material around camera 760 to eliminate infrared light that is 
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reflected off screen 720 without affecting the light that passes through screen 
720. Light reflecting off the screen surface tends to be strongly polarized parallel 
to the screen surface, so a polarizer that encircles camera 760 in a manner 
perpendicular to the screen surface and with its polarization perpendicular to the 
5 screen surface should eliminate most or all stray reflected light from infrared 
illuminator 750 behind screen 720. 



Dealing with Distortion 

[0095] The problem of camera view distortion may be present in all self- 
10 contained interactive video displays described herein (e.g., projection systems 
and transparent flat-panel display-based systems). In many cases, the camera's 
two-dimensional view of the area above the screen may have strong distortion. 
For example, in Figure 7A, object 712 and object 714 are seen as being in the 
same location by camera 760 even though they are at very different positions in 
1 5 front of screen 720. In order for interactions above screen 720 to feel accurate, 
this distortion would need to be corrected. 



[0096] In a distortion-free environment, the virtual position on screen 720 
corresponding to a physical object is the perpendicular projection of that object's 
20 outline onto screen 720. Flat corrective optics such as Fresnel lenses can be 
placed on or near screen 720 so as to redirect the incoming light that is 
perpendicular to the screen toward camera 760. Thus, camera 760 views 
objects in their correct position relative to the surface of screen 720. Figure 8A 
shows an exemplary embodiment of this configuration for a projector-based 
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interactive video display system 800 in cross-section. Camera 810 is placed at 
the focal distance of Fresnel lens 820, causing light rays that shine onto screen 
830 from a perpendicular direction to be redirected at camera 810. As a result, if 
an object moves from position 802 to position 804, its apparent position to 
camera 810 may not change. Thus, a desired effect is achieved, that is, an 
object above screen 820 has a virtual position that is the perpendicular projection 
of the object's outline on screen 820. Note that the optics of the camera's lens 
deserve special consideration; a pinhole lens gives ideal depth of focus and 
image clarity, while a wide-angle lens with the ability to focus past infinity would 
allow a brighter image at some expense to depth of focus and clarity. It should 
be appreciated that the Fresnel lens method of eliminating distortion may be 
used with both projected and transparent flat-panel display based interactive 
systems. 

[0097] In the case of using the Fresnel lens with a self-contained projector 
display, the Fresnel lens does not affect the projected image because the IR- 
transparent, VIS-translucent screen in front of it scatters the projector's light, and 
the distance between this screen and the Fresnel lens is almost zero, so there is 
no distortion of the projected image. 

[0098] In the case of using the Fresnel lens with a transparent flat-panel display, 
the transparent flat-panel display may be placed in front of the IR-transparent 
VIS-translucent screen and Fresnel lens, closest to the viewer. The Fresnel lens 
would not affect the illumination of the display because the white light 
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backlighting is either already diffused or is diffused by a material between the 
Fresnel lens and transparent flat-panel display. 

[0099] Alternatively, the distortion may be eliminated by using a series of mirror 
5 strips on the back of the display, as shown in Figure 8B. These mirror strips 910 
are designed to redirect light shining perpendicularly onto the display toward 
camera 920. The camera is located to the side of the display so as not to 
interfere with the light. The actual number of mirror strips 91 0 may be very large 
and the strips themselves may be very thin. In one embodiment, enough space 
10 between the mirror strips 910 to allow lights from the back of screen 930 to shine 
through is provided. However, camera 920 cannot see these lights; due to its 
perspective, camera 920's view of this area is entirely of the mirror strips 910. 
When viewed in a direction perpendicular to screen 930, mirror strips 910 form 
circular curves in which the center of each circle is at the position of camera 920. 

15 

[0100] Since a projector cannot easily shine through the mirror strips 910, the 
present embodiment may be more usable with a transparent flat-panel display or 
an off-axis projection display in which the projector projects through the space 
between the mirror strips and the screen onto the screen. However, this does 
20 not preclude the use of a projector shining through the mirror strips; although 
some light will be lost, the projector's light is very unfocused at this point, and 
thus the final projected image may be unaffected. 
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[0101] Alternatively, if depth information about the scene being viewed (the 
distance from the camera to the object seen in each pixel) can be obtained, then 
the x-y-z coordinates of the position above the screen occupied by every pixel of 
every object viewed by the camera can be reconstructed (through a simple 
5 coordinate transform) and thus the camera's distorted view can be corrected. 
Such depth information can be obtained using a variety of means, including but 
not limited to stereo cameras, time-of-flight cameras, and patterned illumination. 

[0102] The ability to reconstruct an undistorted three-dimensional (3D) view, 
with x-y-z coordinates for each pixel of the camera's image, would also allow the 
combination of data from multiple cameras into a single unified view of the 
objects in front of the screen by simple superposition of data. The use of 
multiple cameras (or multiple pairs of cameras if stereo vision is used for 3D) 
would allow the display to be made even flatter, with a narrower field of view for 
the cameras. In this case, the cameras or pairs of cameras would be ideally 
placed in a way such that they evenly cover the area behind the screen. For 
example, the cameras may be placed in a grid behind the screen. 

Image Projection and Capture Using Scattering Polarizer Screen 
20 [0103] Another screen material will now be described. This screen material 
serves as an alternative to the "IR-transparent VIS-translucent screen" used in 
the projector-based and transparent flat-panel display-based systems described 
earlier in the current application. Since the screen is disposed between the 
objects and the interactive display system, the screen should transmit the 
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projected image while allowing the illumination source and camera to see 
through the screen clearly. In one embodiment, the screen acts as a 
transmission diffuser for the display image channel while behaving as a window 
for the camera capturing channel. An additional requirement for the screen is to 
5 prevent viewer from glare, that is, the uncomfortable visual stimulus from the 
direct or insufficiently scattered projector light. 

[0104] In certain instances, while small particle scattering has a ^wavelength 
dependence for single-particle scattering, most diffusers employ multiple 

10 scattering to effect adequate diffusion. In multiple scattering the wavelength 

dependence of the scattered light is far more neutral, as is demonstrated by the 
color of milk. The intensity of light coherently scattered by small particles is also 
known to be proportional to (n-1) 2 , where n is the relative index of refraction 
between the particle and the host matrix. In the present invention it is this 

1 5 property of scattering that is exploited to allow transparency in the capture 

channel and haze in the display channel. The method used here is compatible 
with but is not limited to the use of IR in the capture channel. The dispersion of 
normal polymer materials is inadequate to create contrast between the visible 
and near IR. Instead, we label the capture channel with one state of polarization 

20 state and the display channel with the orthogonal state. We also employ a 
screen which has the property of index matching (n=l) between the matrix and 
the dispersed particles for one state of polarization and index mismatching (n?M) 
for the orthogonal state. In this way, the screen will have substantial haze for the 
display channel and substantial transparency in the capture channel. Materials 
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may be tuned to effect a near-perfect index match at the capture channel which 
may have a very narrow spectrum (20 nm or so). Two primary metrics can be 
used to define the performance of this type of screen: the single piece 
transmission (Tsp) and the polarizer efficiency (PE). These quantities are 

i 

5 defined as follows in Equations 1 and 2: 

T sp =(T|,+Tx)/2 (1) 
PE = |(T, r Tx)/(T,|+Tx)| (2) 

10 

where Tj| and T± are the direct (e.g., unscattered or small-angle scatted) 
transmittances in for the two states. 

[0105] For a perfect polarizer T sp =0.5 and PE=1 . For a real polarizer as the 
15 screen thickness or particle concentration increases over a certain useful range, 
T sp will decrease and PE will increase due to multiple scattering. These two 
performance metrics can be optimized for a given application by adjusting the 
materials and processing parameters of a given scattering system. A high T sp 
leads primarily to greater resolution at the camera and a high PE leads primarily 
20 to lower glare. 

[0106] In one embodiment the projector light is polarized in the translucent state 
of the screen. For an LCD projector this can be arranged with very low loss. 
Part of the projected image light will be backscattered and part will be 
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preferentially scattered in the forward hemisphere. The illumination used for the 
camera is preferentially polarized to avoid stray light. This can easily be 
accomplished with a film-type absorption polarizer. Illuminated objects will 
diffusely reflect the light, and roughly equal portions will be produced in the 
5 polarization states corresponding to transparency and scattering (e.g., 

polarization will not be preserved). The camera is fitted with an absorption-type 
polarizing filter so that only the direct light from the object is imaged. The 
camera may also be fitted with a narrow-band filter matching the lamp spectrum 
to avoid video feedback as well as interference from ambient light. 

10 If the scattering polarizer has the further property that the scattered light 

maintains the polarization of the incident light then ambient light reflections will 
be reduced, resulting in higher contrast. Figures 9A and 9B illustrate a 
schematic layout of an interactive video display system having a scattering 
polarizer screen, in accordance with an embodiment of the present invention. 

1 5 The cross and bi-directional arrows indicate states of polarization. 

[0107] A typical LCD projector emits polarized light from its projection lens. 
However, in the case of the most common type of triple panel LCD projector, the 
polarization of the green primary is orthogonal to that of the red and blue 
20 primaries. (This is a result of the X-cube combiner design). Therefore, in order 
to project all three primaries in the same polarization the green must conform to 
the other two. This can be achieved with very low loss by employing a retarder 
stack (available, for example from Polatechno Corp. of Japan), which adds a 
half-wave of retardance to the green channel relative to that of the red and blue. 
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This stack component can be deployed between the combiner cube and 
projection lens or between the lens and the screen. In order to maintain high 
lumen output and avoid image artifacts it is necessary to use a projection lens 
assembly which is polarization preserving. 

5 

Alternative Configuration of Self-Contained Interactive Projected 
Display 

[0108] Screen materials that are partially transparent but translucent when light 
is shined on it at a particular angle, such as the HoloClear, a holographic screen 

10 manufactured by Dai Nippon Printing, can be used in the self-contained 

interactive display in a way that the interior of the display container is completely 
dark to the user. This can be accomplished by making all inside faces of the 
container black, with a black window that is transparent to infrared in front of the 
infrared camera and illuminator. Since the screen material is partially 

15 transparent, the camera is able to see objects beyond it. However, since the 
projector is offset at the appropriate angle (e.g., 35 degrees in the case of 
HoloClear), the light from the projector is completely diffused, eliminating the 
glare. Users of the display may not see anything behind the partially transparent 
screen because the interior is completely blacked out. 

20 

[0109] In another embodiment, a screen material can be used that can switch 
from clear to translucent almost instantly when an electric current is applied, 
such as the "Privacy Glass" product currently being marketed to interior 
designers. This material is referred to herein as a time-based material (e.g., 
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transparent or translucent dependent on time). This material may be used 
instead of the wavelength or polarization selective screen. The camera 
exposures are very brief (e.g., approximately 100 microseconds, 30 times a 
second). When the camera is exposing, the screen material turns clear, allowing 
5 the camera to see through the screen. In the case of a projector system, an 

electronic (e.g., high speed liquid crystal shutter) or mechanical shutter can block 
the projector's light output, ensuring that the projector does not shine in the 
user's eyes during this time. When the camera is not exposing, the screen 
material turns translucent, allowing the projector's or backlight's light to be 
10 scattered. It should be appreciated that the term backlight refers to the 
illumination source for illuminating the flat-panel display with visible light. 



General Description of Interface 

[0110] Although described implicitly by the text herein, this system describes an 
15 interface to a self-contained display system. This interface allows the sensing of 
the position, outline, and potentially the distance of objects (including human 
users) in front of the display, and the display of real-time effects based on this 
sensing. These real-time effects may include a mapping of actions in the 
physical space in front of the display to effects of those actions in a 
20 corresponding location in the virtual space of the display. In other words, the 
system may be calibrated so that interaction with a virtual object on the display 
happens when a physical object such as a user's hand is placed at the location 
of the virtual object on the display. This display has the physical property that 
there is no visible sign of any sensing equipment; only the display screen itself is 
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visible. In the case of a window display system, described later in the present 
application, the interface is the same as described for the self-contained display, 
but the display takes the form of a window display in which no apparatus is 
placed on the same side of the window as the user. 

5 

[01 11] In a number of embodiments of the present invention, a video camera is 
used as the sensing apparatus. The camera's images serve as input to a 
computer vision system that separates foreground objects (such as people) in 
the camera's images from static background in real-time. This foreground- 

10 background distinction serves as an input to an interactive video display 

application that generates the images that are displayed on the display screen. 
These images can be calibrated such that the effects of an object on the 
displayed image of the interactive application are in the same physical location 
as the object. This creates the illusion of an augmented reality, in which a 

15 person can interact with images or objects on the screen through natural body 
motions such as picking up, pushing, or pulling, allowing the illusion of 
manipulating real objects or images 

Transparent Display Screen 
20 [01 12] In one embodiment of the system, an LCD screen or other such 

transparent screen is used as the display apparatus. The camera is used for 
sensing the motion of human users is placed behind the screen. Thus, the 
camera views the area in front of the screen by looking through the screen. This 
area in front of the screen, where human users and objects can be detected by 
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the camera, is called the interactive area. Therefore, the screen is at least 
partially transparent to the wavelengths of light viewed by the camera. 

[01 13] In order to prevent the content being displayed on the screen from 
5 affecting the camera's image of objects beyond the screen, the camera operates 
at a wavelength of light for which the screen is partially transparent no matter 
what content (including the color black) is being displayed. Ideally, the content 
on the screen should have no effect on the optical properties of the screen at the 
wavelengths of light viewed by the camera. In the case of the LCD monitors 

10 used in laptops and flat-panel computer displays, the LCD screen typically 
achieves this property when the camera viewing through it is only sensitive to 
wavelengths of light of 920nm or longer. However, a few LCD panels also 
achieve this property at wavelengths closer to visible light, such as 800nm. In 
addition, the polarizers in the LCD screen do not polarize light at these 

1 5 wavelengths. 

[01 14] The LCD or other transparent display screen is illuminated so as to allow 
the viewer to see the content on it. Ideally, this light should be bright and evenly 
spread across the screen. Typical LCD displays use any one of a variety of 
20 back-lighting and edge-lighting solutions. However, because these solutions 

typically involve putting several layers of scattering, reflective, or opaque material 
behind the screen, they do not allow a camera behind the screen to view the 
area in front of the screen. However, the present invention describes several 
selective scattering materials that allow the camera to view the area in front of 
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the screen while still providing bright and even illumination of the area in front of 
the screen. 

[01 15] In the following solutions, the illumination source for backlighting or edge 
5 lighting is preferably a long-lifetime efficient white visible light emitter, such as, a 
fluorescent lamp or white LED, but can be any source of visible light. 

1 . Rayleigh scattering material 

[01 16] One solution involves placing a sheet of material with strong Rayleigh 
10 scattering on the back surface of the screen, using white backlighting or edge 
lighting to illuminate the display screen, and using a near-infrared-sensitive 
camera to view through the screen. Since Rayleigh scattering is proportional to 
the inverse fourth power of the wavelength of light being scattered, almost all the 
white light is scattered by the Rayleigh material, providing even illumination of 
1 5 the screen. However, relatively little of the infrared light viewed by the camera 
will be scattered because infrared is of a longer wavelength than visible light. 

2. Textured material 

[01 17] Another solution involves creating a flat sheet of material with a physical 
20 texture of bumps, ridges, pits, or grooves interspersed between flat areas. This 
material can then be placed on the back surface of the display screen. This 
material can have the effect of scattering all light that passes through it at a 
glancing angle, while only scattering a small portion of light that passes through it 
perpendicularly. Some such materials are shown in Figures 10A and 10B. 
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Figure 10A shows a simplified cross-section of a material 1000 that has 
microscopic bumps or ridges 1010 that scatter light. The scattering may be 
accomplished by texturing the surface of the ridges or bumps 1010, by making 
the ridges or bumps out of a material that scatters light, or through other 
5 methods. Figure 10B shows a simplified cross-section of a material 1050 that 
has microscopic grooves or pits 1060 that scatter light. The scattering may be 
accomplished by texturing the surface of the grooves or pits 1060, by filling in the 
grooves or pits 1060 with a material that scatters light, or through other methods. 
In all cases, a significant portion of light that passes through near perpendicular 
10 to the surface of the material will not be scattered, while nearly all of the light that 
passes at a glancing angle to the surface will be scattered. 

[0118] Thus, if the screen is illuminated through edge lighting, the display screen 
can be evenly and brightly lit while allowing the camera to see through the 

15 screen. Figure 1 1 illustrates a simplified schematic diagram of a self-contained 
edge-lit interactive display in cross section, in accordance with an embodiment of 
the present invention. Edge lighting 1110 provides visible illumination for 
illuminating display screen 1140 (e.g., an LCD). Screen 1150 is placed adjacent 
to display screen 1 140, and is operable to scatter light that hits it from a glancing 

20 angle (e.g., the angle of edge lighting 1110. Illuminators 1 1 20 illuminate object 
in the field of view of camera 1 130. Light from illuminators 1 120 hits display 
screen 1 140 at a perpendicular or near-perpendicular angle, and is not 
scattered. 
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3. Scattering polarizer 

[01 19] In another embodiment, a scattering polarizer, as described in the 
section "Image Projection and Capture Using Scattering Polarizer Screen" is 
placed on the back of the display screen. This scattering polarizer scatters light 
5 of one polarization, while leaving light of the opposite polarization unscattered. 
The display screen can be evenly illuminated using backlighting by linearly 
polarizing the backlight in the same direction as the scattering direction on the 
scattering polarizer. Thus, all of the backlight is scattered before it passes 
through the display screen. 

10 

[0120] Alternatively, the backlight can be unpolarized and a linear polarizer may 
be placed between the scattering polarizer and the display screen, with the 
polarizer's polarization oriented in the same direction as the direction in the 
scattering polarizer that scatters the light. Thus, any light from the backlight that 
15 is not scattered by the scattering polarizer is of opposite polarization to the linear 
polarizer, causing it to be absorbed. This causes the illumination on the display 
screen to be even and annoying glare in the user's eyes is eliminated. 

[0121] If the display screen is an LCD screen, then the backlight does not need 
20 to be polarized because there is a linear polarizer built in to the back surface of 
the LCD screen. In this case, even illumination can be achieved from an 
unpolarized backlight simply by placing the scattering polarizer on the back of the 
LCD screen, oriented such that the direction of maximal scattering in the 
scattering polarizer is parallel to the polarization of the linear polarizer on the 
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backside of the LCD screen. Thus, only light that is scattered by the scattering 
polarizer is allowed to pass through the LCD screen. 

Flat-Panel Display Screen 
5 [0122] A simplified cross-sectional diagram of an exemplary embodiment of a 
self-contained display 1200 utilizing a display screen is shown in Figure 12A. 
The display is created using an LCD screen 1210 that is backlit by a white visible 
light 1220. The scattering polarizer 1215 scatters all this light, providing even 
illumination for the viewer. Mirrors 1225 on the side of the self-contained unit 

1 0 reflect stray white light from the lights 1 220 back towards the display screen 
1210, increasing its brightness. A video camera 1230 sensitive to only near- 
infrared light from 920nm to 960nm views the area in front of LCD screen 1210, 
referred to as the "camera's field of view". Objects within this field of view will be 
visible to camera 1 230. Illumination for the camera's field of view comes from 

1 5 sets of infrared LED clusters 1 240, which produce light in wavelengths viewable 
by camera 1230, on the back side of the box. The light from these LEDs 1240 is 
slightly scattered by a diffusing screen 1245 before it reaches LCD screen 1210 
to prevent bright specular highlights from the LEDs 1240 from showing up on the 
camera's image. A Fresnel lens 1250 is used to reduce the distortion of the 

20 camera's view of the area in front of LCD screen 1210. 

[0123] The paths of visible and infrared light through the exemplary embodiment 
in Figure 12A will now be described. We will refer to the two perpendicular 
polarizations of light as polarization A and polarization B. 
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[0124] Visible light from the white light illuminators 1220 starts unpolarized, and 
may be scattered by a diffusing material 1245, redirected by a Fresnel lens 1250, 
or reflected off the mirrors 1225 on its path toward screen 1210. Next, this light 
5 passes through scattering polarizer 1215, which scatters all the light of 

polarization A and none of the light of polarization B (where A and B refer to two 
perpendicular polarizations). The scattered light retains its polarization after 
being scattered. This light then passes through LCD screen 1210, which 
absorbs all the light of polarization B and transmits all the light of polarization A. 
10 Thus, LCD screen 1210 is illuminated using only scattered light, and the viewer 
sees an evenly illuminated screen. 

[0125] The infrared light emitted from the infrared illuminators 1240 may begin 
unpolarized. Optionally, for improved clarity, this light can first pass through an 

15 infrared linear polarizer 1260 to polarize it in polarization B so that less of it will 
be scattered by the scattering polarizer 1215. Next, the infrared light may be 
scattered by a diffusing material1245, redirected by a Fresnel lens 1250, or 
reflected off the mirrors 1225 on its path toward screen 1210. If the light is 
unpolarized, some of it will be scattered as it passes through scattering polarizer 

20 1215, but the light of polarization B will pass through scattering polarizer 1215 
unscattered. Since the wavelength of the infrared light is sufficiently long, it 
passes unaffected through LCD screen 1210 and can illuminate objects in front 
of the screen, such as a human hand. 



49 



WO 2005/057399 



PCT/US2004/041320 



[0126] Infrared light returning from in front of the display screen toward camera 
1230 will be unaffected by LCD screen 1210. However, as the light passes 
through scattering polarizer 1215, the light of polarization A will be scattered 
while the light of polarization B will remain unscattered. Next, the light passes 
5 through Fresnel lens 1250, but it does not significantly affect polarization. 

Camera 1230 has an infrared linear polarizer 1260 immediately in front of it; this 
polarizer 1260 absorbs light of polarization A and transmits light of polarization B. 
Thus, camera 1230 only views light of polarization B, which was left unscattered 
by scattering polarizer 1260. This gives camera 1230 a clear, high-contrast 
10 image of the area in front of the screen. 

[0127] Another exemplary embodiment of an LCD-based interactive display is 
shown in cross section in Figure 12B. The overall system is wedge-shaped. The 
system design is similar to the design as shown and described around Figure 

15 12A. However, the infrared illuminators have been positioned to minimize glare 
into the camera 1262. Objects on and near the screen are illuminated by the 
interior infrared illuminators 1264, which shine through the scattering polarizer 
1266 and the LCD panel 1268. However, they do not shine through Fresnel lens 
1276 in order to reduce glare effects on the camera 1262. The Fresnel lens 

20 1276 is set back from the surface of the LCD panel 1268 and the scattering 
polarizer 1266 in order to provide room for interior infrared illuminators 1264. 
Exterior infrared illuminators 1270 illuminate objects that are further away from 
the screen. Infrared illuminators 1270 shine around (rather than through) LCD 
panel 1268 or scattering polarizer 1266, allowing glare to be further reduced. 



WO 2005/057399 



PCT/US2004/041320 



The white visible light illuminators 1272 are arranged along the sides of the 
system's base, and are covered by backlight cover 1274. Backlight cover 1274 
may consist of a material that absorbs near-infrared but transmits visible light in 
order to reduce the presence of ambient infrared on the screen, and therefore 
5 improve the contrast of the image captured by camera 1262. 

Projected Display Screen Using a Scattering Polarizer 
[0128] In another embodiment of the interactive video display system, a 
projector and projection screen are used as the display apparatus. The camera 
10 Used for sensing the motion of human users is placed behind the screen. Thus, 
the camera views the area in front of the screen by looking through the screen. 
Therefore, the screen is at least partially transparent to the wavelengths of light 
viewed by the camera. The scattering polarizer can serve as the projection 
screen in this system. 

15 

[0129] It should be appreciated that the scattering polarizer may not operate 
perfectly. A small amount of light in the polarization that is supposed to be 
scattered may not be scattered. Because of the extreme brightness of projector 
light when it is viewed directly, the bright light source inside the projector's lens 
20 may still be directly visible through the scattering polarizer, even though the 
projector's light may be polarized in the appropriate direction for maximum 
scattering. This bright spot of glare can be eliminated by using a linear polarizer 
in front of the scattering polarizer in order to ensure that unscattered projector 
light is absorbed. In addition, if the projector's light is not completely polarized, a 
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similar problem will appear. This problem can be reduced by using a linear 
polarizer behind the scattering polarizer. In both cases, these polarizers are 
oriented parallel to the projector's light. If the camera is near-infrared or another 
non-visibfe light wavelength, the visible-light polarizer can be chosen such that 
5 the camera is unaffected by it. Thus, the camera is able to view through the 
screen. 



ELIMINATING SPECULAR REFLECTIONS 

[0130] In addition, specular reflections from the camera's illuminators can 
10 adversely affect the camera's image. These effects can be mitigated by applying 
antireflective coatings to one or both surfaces of the display screen as well as 
any surfaces behind the screen, including the Rayleigh scattering material, 
textured material, scattering material, or Fresnel lens. These effects can also be 
mitigated by angling the light coming out of the illuminators so that there is no 
1 5 specular reflection from the camera's illuminators back into the camera. 

[0131] One example of such a configuration is shown in Figure 13. This 
configuration employs spot illuminators 1310 that are distant from camera 1320 
and shine perpendicular to screen 1330, preventing any specular reflection into 
20 camera 1320. Areas not covered by these illuminators are lit by illuminators 

1340 that shine at a glancing angle to screen 1330, preventing the reflected light 
from shining back to camera 1320. 
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[0132] In another embodiment, the scattering polarizer 1410 or other selective 
scattering material can be slightly bent so that specular reflections of the 
camera's illuminators 1440 are bounced away from camera 1430, as shown in 
Figure 14. As shown, specular reflections from illuminators 1440 are redirected 
5 toward the sides of the box. In another embodiment, a diffusing material can be 
placed in front of the camera's illuminators to soften any specular reflections into 
camera. Light from these illuminators could also be diffused by bouncing their 
light off the back of the display. 

10 [0133] Figure 15 illustrates an exemplary configuration of an interactive video 
display system 1500 using time-of-flight cameras 1530, in accordance with an 
embodiment of the present invention. Specular reflections pose a particular 
problem for time-of-flight cameras 1 530 because, in typical designs, the camera 
and the camera's illuminators must be placed immediately adjacent to each 

15 other, and the light from illuminators cannot be scattered. Thus, the 

aforementioned approaches will not work in an implementation that uses time-of- 
flight cameras due to the camera's illuminator causing severe glare from its 
reflection off of the screen back into the camera. However, because the 
computer vision system can use the 3D camera information to perform a 

20 coordinate transform to the desired coordinate system, the camera need not be 
placed behind the center of the screen, and data from multiple cameras could be 
merged. Thus, for example, two time-of-flight cameras 1530 could be used to 
view the area in front of screen 1 51 0 at an angle so as to avoid any specular 
reflection from their illuminators, as long as the two cameras 1530 took their 
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exposures at different times. In this configuration, neither camera would be able 
to view the specular reflection of its built-in illuminator. 

Adding a Touchscreen Interface 
5 [0134] Although the described system is able to recognize objects and gestures 
several inches away from the screen, touchscreen behavior can be provided as 
well, in which users have to literally touch the screen to cause an action to take 
place. This will allow the system to support additional varieties of user 
interfaces. For example, this interface could allow users to "gather" virtual 
10 objects on the screen by cupping their hands together above the screen, but also 
"pick up" a virtual object by touching its image and then "drop" the object by 
touching another point on the screen. 

[0135] This touchscreen behavior can be implemented in one of several ways. 

15 The touchscreen examples described in this section and subsequent sections 
are compatible with both projector-based and transparent flat-panel display 
based interactive video systems. An existing touchscreen technology can be 
integrated with the system's screen so long as the portion covering the display 
screen is transparent to the camera. This includes resistive touchscreens, 

20 capacitive touchscreens, infrared grid touchscreens, and surface acoustic 

touchscreens. However, all of these screens have the drawback that they can 
only detect one finger touching the screen at a time; some cannot even detect a 
continuous touch as opposed to a brief tap. There are several solutions which 
overcome the foregoing drawback, many of which also allow the system to 
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collect information about the distance between an object and the screen. This 
3D data is useful for higher-level vision processing such as gesture recognition. 

Multi-User Touchscreens and 3D Data: Stereo Camera 
5 [0136] A multi-user touchscreen can be created, which allows multiple people to 
use the screen simultaneously, by making some slight modifications to this 
system. In one embodiment, a stereo camera can be used in place of a single 
camera. The stereo camera would have the same wavelength sensitivity and 
filter setup as the single camera system. However, the computer could take the 
10 two images from the stereo camera, and, using any one of several well-known 
stereopsis algorithms, such as the Marr-Poggio algorithm, deduce distance 
information for any objects seen on-screen. Since the distance to the screen is 
known, the computer can identify whether any object is touching the screen by 
comparing the object distance information to the distance to the screen. 

15 

Multi-User Touchscreens and 3D Data: One-camera Stereo 
[0137] Figure 16 illustrates a configuration 1600 for using a mirror to get 3D 
information, in accordance with an embodiment of the present invention. Stereo 
data can be acquired using only one camera 1610 by placing a mirror 1620 on 
20 the inside side of the box. Thus, camera 1610 would see screen 1630 both 
directly and at an angle. An object that is touching screen 1630 will appear the 
same distance from the edge of mirror 1620 in both the camera's main image 
and reflected image. However, an object above screen 1630 will be at different 
distances from the edge of mirror 1620 in the main and reflected images. By 
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comparing these images, the computer can deduce whether each object is 
touching screen 1630. 

Multi-User Touchscreens and 3D Data: Patterned Illumination 
5 [0138] Figure 17 illustrates another configuration 1700 for using patterned 
illumination to get 3D information, in accordance with an embodiment of the 
present invention. A patterned infrared illuminator 1710, which projects a light 
pattern, can also be used in place of the regular infrared illuminator. This system 
could distinguish between objects on screen 1720 and above screen 1720. 

10 However, the precision of this system could be improved by having the patterned 
infrared illuminator 1710 shine on screen 1720 at an angle, perhaps by bouncing 
it off a mirror 1730 on the interior side of the box. The position of the patterned 
light on an object would change dramatically as it is moved even a short distance 
away from or towards screen 1720, allowing the distance between the object and 

1 5 screen 1 720 to be more easily determined by computer 1 740. 

Multi-User Touchscreens and 3D Data Time of Flight System 
[0139] The camera could be a time-of-flight infrared camera, such as the models 
available from Canesta and 3DV systems. This camera has a built-in capability 
20 to detect distance information for each pixel of its image. If this sort of camera is 
used, then it will be crucial to eliminate all infrared glare from the screen; 
otherwise, the camera will mistake the illuminator's reflection on the screen for 
an actual object. Methods for eliminating infrared glare are described above. 
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Multi-user Touchscreens: Multiple Wavelengths 

[0140] Touchscreen behavior can also be achieved by having two cameras and 
two illuminators, with each camera-illuminator pair at a different infrared 
frequency. For example, one camera-illuminator pair could use 800 nm light 
5 while the other could use 1 100 nm light. Since the screen's scattering is 

proportional to the inverse fourth power of the wavelength, the 800 nm camera 
would have far less ability to see what is beyond the screen than the 1 100 nm 
camera. As a result, only objects that are touching or almost touching the screen 
would be visible to the 800 nm camera, while the 1 100 nm camera would be able 
10 to see objects several inches away from the screen. Both cameras would input 
their data to the computer and run through separate vision systems. The 800 
nm data would be the input to the touchscreen interface, while the 1 100 nm 
camera would be used for gesture input because it can detect objects above the 
screen. 

15 

Multi-User Touchscreens: Narrow Beam on Display Surface 
[0141] Figure 18A illustrates configurations 1800 and 1850 for using additional 
illuminators to get 3D information, in accordance with an embodiment of the 
present invention. A second illuminator 1810 can be used to light only objects 
20 that are on or very close to the display surface. For example, if a narrow-angle 
illuminator, such as an LED or laser, is placed just inside or outside screen 1820, 
pointed nearly parallel to screen 1820, then objects next to screen 1820 will 
show up very bright in the camera's image. Cylindrical lenses or other means 
may be used to spread the illuminators' light horizontally but not vertically, 
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allowing it to cover the full area of the screen. The vision system can then 
deduce that very bright objects are either very close to or touching screen 1820. 

[0142] In another embodiment, the beam can be shined inside the display 
5 surface. The illumination system of such an embodiment as shown in Figure 
18B. A transparent pane 1860 is placed in front of the main display screen 
1855. An illuminator 1870 shines a narrow beam into the edge of transparent 
pane 1860. Because of the steep angle of incidence, the light 1870 reflects 
completely within the confines of transparent pane 1860. However, if an object 
10 1875 is touching the screen, the light 1870 is able to scatter, allowing it to 

escape the confines of transparent pane 1860. This light 1880 is then able to be 
detected by the camera 1885. This allows the camera 1885 to detect when an 
object 1875 or user is touching the screen. 

15 [0143] For these approaches, which utilize a second set of illuminators at or 

near the surface of the screen, these are several designs which allow the system 
to distinguish between light from the main illuminators and light from the 
secondary illuminators. This distinction is important as it allows the system to 
detect objects touching the screen separately from objects that are merely in 

20 front of the screen. 

[0144] First, the two sets of illuminators could turn on during alternate camera 
exposures, allowing the system to see the area in front of the screen as lit by 
each illuminator. Alternately, the system could use two cameras, with each 
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camera-illuminator pair operating at a different wavelength. In addition, the 
system could use two cameras but have each camera-illuminator pair strobe on 
at different times. 

5 Multi-User Touchscreens and 3D data: Brightness Ratios 

[0145] Touchscreen behavior can be achieved by placing different illuminators 
at different distances to the screen. Suppose that illuminator A is two feet away 
from the screen and illuminator B is one foot away from the screen. The 
brightness of an illumination source is proportional to the inverse square of the 
10 distance from the illumination source. Thus, the ratio of light from A and B on an 
object changes as its distance changes. This ratio allows the computer to 
determine whether an object is on the screen. Table 1 shows an example of 
how the ratio between light from A and B can differentiate between an object on 
the screen and an object even an inch above the screen. 

15 



Object position 


Light from 


Light from 


Ratio of light from 




illuminator A 


illuminator B 


B to light from A 




(relative to 1ft 


(relative to 1 ft 




away) 


away) 




Touching screen 


0.25 


1 


4 to 1 


1 inch above screen 


0.23 


0.85 


3.7 to 1 


1 foot above screen 


0.11 


0.25 


2.3 to 1 



Table 1 



[01 46] The ratio holds true ho matter what color the object is so long as it is not 
completely black. In addition, because the LED light is nonuniform, the ratio of 
20 an object touching the screen may vary depending on what part of the screen the 
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object is touching. However, that ratio can be established during the calibration 
process and reaffirmed overtime by recording the maximum ratios recently 
observed at each point on the screen. 

5 [0147] Illuminator A and illuminator B can be distinguished. In one embodiment, 
two cameras and illuminators are used and tuned to different wavelengths, as 
described above under the heading "Multi-user touchscreens: Multiple 
wavelengths". Thus, illuminator A is only visible to camera A, and illuminator B is 
only visible to camera B. 

10 

[0148] In another embodiment, illuminator A and illuminator B have the same 
wavelength, but are turned on at different times. If there is one camera, 
illuminators A and B can alternate turning on during the camera exposures so 
that all even-numbered camera exposures are taken while A is on and B is off, 

1 5 and all odd-numbered camera exposures are taken while A is off and B is on. 
This lowers the effective frame rate by a factor of 2, but allows one camera to 
capture images lit by A and B separately. The computer can then compare the 
two images to compute the brightness ratio at each point and determine when an 
object is touching the screen. It is easy to synchronize the illuminators to the 

20 camera by creating a circuit that reads or generates the camera's sync signal. 

[0149] In another embodiment, two separate cameras can be linked to 
illuminators A and B (with both at the same wavelength), so long as the 
illuminators are strobed to turn on only when the corresponding camera is taking 
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an exposure, and the exposures of the two cameras are staggered so that they 
do not overlap. 

[0150] In all cases, the computer can compare the images of the screen lit by A 
5 and the screen lit by B to determine the brightness ratio at each point, and thus 
the distance of any objects from the screen. The example with A two feet from 
the screen and B one foot from the screen is simply one embodiment; other 
distance ratios or arrangements also work. 

10 Tiling 

[0151] Because the system is in a self-contained box and the screen can take 
up an entire side of that box, screens can be stacked together in a grid, with all 
their screens on the same side, to create a much larger screen. If the computer 
in each system is networked together, the systems can share information about 
15 their real-time vision signals and content, allowing the screens to function as one 
large seamless screen. For aesthetic reasons, the screens of the individual tiled 
units may be replaced with one very large screen. 

Getting Distance Information From Amount of Blur 
20 [0152] Some screen materials and structures can cause scattering of incident 
light such that the typical angle of scattering is small. Thus, the majority of light 
passing through the screen material changes direction slightly. Figure 19A 
shows a conceptual example of slight scattering; the length of the arrow for each 
scattered ray of light represents the portion of light scattered in its direction; 
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although a finite number of arrows are shown, the distribution of scattering 
angles is continuous. 

[0153] This form of scattering can be achieved in several ways. Materials with 
strong Mie scattering exhibit this property. In addition, materials that have 
textured surfaces also redirect light slightly in the desired manner. Ideally, the 
texture would only cause a small average and probabilistically smooth deviation 
in the light's path. Figure 19B shows an example of such a texture. The size of 
the texture would be small enough to not detract from the viewing of the 
projected image. This scattering effect may be accomplished by altering the 
main screen material to have either strong Mie scattering or a textured surface. 
Alternately, these properties may be added to a second screen material that is 
sandwiched together with the main screen material. 

15 [01 54] The use of such a material as part of the screen would affect the way the 
camera views the scene. All objects viewed by the camera would be blurred due 
to the material's scattering property. However, the amount of blurring would 
depend on the distance. Objects touching the screen would remain unblurred, 
but ones further away would be progressively more blurred. This is because 

20 scattering a light ray by a given angle at the screen's surface translates to a 
physical distance of scattering that is proportional to the distance from the 
screen to the light ray. For example, scattering a light ray horizontally by 45 
degrees causes it to deviate in its horizontal position by 1 foot at a distance of 1 
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foot from the screen, but the deviation is 2 feet at a distance of 2 feet from the 
screen. 

[0155] The vision system can then use this blurred image to reconstruct the 
5 distance by many methods, including the use of edge detection techniques that 
detect both sharp and blurry edges and can make estimates of the amount of 
blur for each edge. The Elder-Zucker algorithm is one such edge detection 
technique. Once the amount of blur is known, the distance of the object's edge 
from the screen can be determined, giving the vision system 3D information 
10 about that object since the amount of blurriness is proportional to the distance. 

[0156] The task of the vision system can be simplified by using a patterned 
illumination source, which projects an infrared pattern visible to the camera, 
instead of or in addition to the regular infrared illuminators. This pattern may 
15 include dots, lines, or any other texture with sharp edges. The pattern may be 
projected from several directions, including through the screen. If the pattern is 
projected through the screen, the amount of scattering will be doubled, but the 
effect of distance-dependent scattering will not change. 

20 [0157] By illuminating all objects on the screen with a pattern, the vision 
system's performance is increased. Without the patterned illumination, it is 
difficult to determine 3D information at any location in the image where there are 
no edges, such as the middle of an object of uniform brightness. However, with 
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all objects covered in this projected pattern, it is easy to get blur information at 
any point in the image. 

[0158] With a projected texture, different methods can be used to estimate the 
5 amount of blur. Image convolutions such as Sobel filters or pyramid 

decompositions can be used to get information about signal strengths and 
gradients on different scales. If the texture is a dot pattern, places in the image 
that correspond to the dots can be located by looking for local maxima. Then, by 
examining the strength of the gradient of the area around each local maximum, it 
10 can be determined how much blurring has taken place. The gradient at the edge 
of the dot is roughly inversely proportional to the amount of blurring. Hence, the 
gradient at each dot can be related to the distance from the screen. 

Camera Configurations 

1 5 [01 59] In one embodiment of the system, the camera is sensitive to light of a 
wavelength that is not visible to the human eye. By adding an illuminator that 
emits light of that invisible wavelength, the camera can take well-illuminated 
images of the area in front of the display screen in a dark room without shining a 
light in the users' eyes. In addition, depending on the wavelength of light 

20 chosen, the content of the display screen may be invisible to the camera. For 
example, a camera that is only sensitive to light of wavelength 920nm-960nm will 
see an LCD screen as transparent, no matter what image is being displayed on 
it. 
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[0160] In another embodiment of the system, the camera is only sensitive to a 
narrow range of wavelengths of near-infrared, where the shortest wavelength in 
the range is at least 920nm. The area in front of the screen is illuminated with 
clusters of infrared LEDs that emit light in this wavelength range. The camera is 
5 a near-infrared-sensitive monochrome CCD fitted with a bandpass filter that only 
transmits light of the wavelengths produced by the LEDs. For further image 
quality improvement and ambient light rejection, the camera and LEDs can be 
strobed together. 

10 [0161] In one embodiment of the system, the camera has a relatively 

undistorted view of the area in front of the screen. In order to reduce distortion, 
the camera can be placed at a significant distance from the screen. 
Alternatively, the camera can be placed closer to the screen, and a Fresnel lens 
can be placed on or behind the screen. Figure 20A shows a high distortion 

15 configuration, with the camera very close to the screen. Note that in Figure 20A, 
Object 2010 and Object 2020 appear to be in the same position from the 
camera's perspective, but are over very different parts of the screen. Figures 
20B, 21 A and 21 B show several configurations in which distortion is reduced. 
Figure 20B shows a low distortion configuration in which the camera is far from 

20 the screen; the overall display can be kept compact by reflecting the camera's 
view. Note that in Figure 20B, Object 2030 and Object 2040 appear to be in the 
same position from the camera's perspective, and occupy similar positions 
above the screen. Figures 21 A and 21 B show the use of Fresnel lenses to 
reduce or eliminate distortion respectively. 
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[0162] Fresnel lenses can also be used to allow multiple cameras to be used in 
the system. Figure 21 C shows the use of Fresnel lenses to eliminate distortion 
in a two-camera system. Each camera has a Fresnel lens which eliminates 
5 distortion for that camera's view. Because the fields of view of the two cameras 
just barely touch without intersecting, objects will be able to pass seamlessly 
from one camera's view to the other camera's view. This technique extends to 
larger numbers of cameras, allowing a grid of cameras to be placed behind the 
screen. This technique allows the interactive display to be very shallow, giving it 
10 a form factor similar to a flat-panel display. In a similar way, the use of a Fresnel 
lens to eliminate distortion allows multiple self-contained displays to be tiled 
together in a way that allows the cameras from all the displays to be seamlessly 
tiled together. 

[0163] If a technique is used to acquire a 3D image from the camera, the 
camera's position becomes less important, as the distortion can be corrected in 
software by performing a coordinate transformation. For example, the camera's 
depth reading for each pixel can be transformed to an (x,y,z) coordinate, where x 
and y correspond to a position on the display screen nearest to the point, and z 
corresponds to the distance from the position (x,y) on the screen to the point. A 
3D image can be obtained in hardware by using a time-of-flight camera, among 
other software-based and hardware-based approaches. Manufacturers of 3D 
time-of-flight cameras include Canesta and 3DV Systems. The aforementioned 
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approaches are fully compatible with placing a time-of-flight camera behind the 
screen, since most time-of-flight cameras use infrared illuminators. 

Illuminators for the Camera 
5 [0164] Illuminators that illuminate the interactive area in front of the screen in 
light of the camera's wavelength can be placed either around the screen, behind 
the screen, or both. 

[0165] If these illuminators are placed around the screen, they shine directly 
1 0 onto the interactive area, allowing their brightness to be put to maximal use. 
However, this configuration is unreliable; users may block the illuminator's light 
path, preventing some objects in the interactive area from being illuminated. 
Also, this configuration makes it difficult to illuminate objects that are touching 
the screen. 

15 

[0166] The aforementioned problems with illuminators placed around the screen 
are solved by placing illuminators behind the screen; with illuminators near the 
camera, any object visible to the camera will be illuminated. However, the light 
from these illuminators may be backscattered by the Rayleigh scattering 
20 material, textured material, or scattering polarizer behind the screen. This 

backscattering significantly reduces the contrast of the camera's image, making it 
more difficult for the vision system to decipher the camera's image. 
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[0167] If the light is being scattered by a scattering polarizer, the camera is 
sensitive to near-infrared light, and the illuminator emits near-infrared light, then 
the aforementioned contrast loss can be reduced through the use of infrared 
linear polarizers, which linearly polarize infrared light. Placing an infrared linear 
5 polarizer in front of the camera, with the polarization direction parallel to the 
direction at which the scattering polarizer is transparent, will significantly reduce 
backscatter and improve contrast. Placing an infrared linear polarizer in front of 
the infrared illuminator, with the polarization direction parallel to the direction at 
which the scattering polarizer is transparent, will also reduce backscatter and 
10 improve contrast. 



Window Display 

[0168] According to another aspect of the present invention, the self-contained 
interactive video displays can be used with a window display. Self-contained 
15 interactive video displays can be deployed in a variety of physical configurations, 
for example, placing the screen horizontal, vertical, or diagonal. However, when 
deploying such a display on a window, there are several additional possible 
physical configurations. 



20 [0169] Figure 22 illustrates an exemplary configuration of an interactive window 
display 2200, in accordance with an embodiment of the present invention. In 
one embodiment, instead of being self-contained, the components can be 
physically separated. The screen 2210 can be affixed directly to the window 
2220 surface or mounted separately behind the window 2220. The camera 
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2230, projector 2240, and illuminators 2250 can be placed either in nearby or 
separate locations, and may be mounted on the floor, ceiling, or anywhere in 
between at various distances from the window 2220. Optionally, the infrared 
illuminators 2250 can be placed to the side of the screen 2210 so that they shine 
directly onto the subject instead of through the screen 2210. Also optionally, the 
communication between the camera 2230 and computer 2260, or between the 
computer 2260 and projector 2240, may be wireless. 

[0170] The camera in window displays is generally aimed horizontally. 
Consequently, the camera usually views people at an arbitrary distance from the 
screen. While the vision software, screen material, or other systems can be 
used to identify and remove objects at an excessive distance, it is also possible 
to tilt the camera upwards so that more distant objects may need to be of a 
certain minimum height in order for the camera to see them. Thus, only people 
within a few feet of the screen are able to interact with it. Users approaching 
such a display will notice their virtual presence first appear at the bottom of the 
screen and then gradually rise up as they come closer to the screen. Figure 22 
shows a camera 2230 tilted upward in this manner. 

[0171] Glare is an issue in window displays. However, users of a window 
display are limited in the angles that they typically to view the screen. They will 
be unlikely to look at the screen from an oblique angle in general because they 
will probably maintain a distance of at least a few (e.g., two) feet from the display 
so as to have room to point at objects on the display with their arms and hands. 
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For a display at and below eye level, people are especially unlikely to look up at 
the display from an oblique angle. If the projector is placed extremely close to 
the screen but gbove the top of the screen, with its beam projecting downward at 
an oblique angle, then this low-glare situation will be realized. Alternatively, if the 
5 display is set up to be at or above eye level, then a similar glare reduction can be 
achieved by placing the projector below and close to the screen, projecting 
upward at an oblique angle. Note that off-axis projectors are especially useful for 
these sorts of configurations. 

10 Window Unit: Alternative Configuration 

[0172] Visible-light transparency of the screen is more desirable in a window unit 
than in a self-contained unit. Thus, window displays can be built with a partially 
transparent screen that is translucent to light that is shined on it at a particular 
angle. One material that can be used to build the partially transparent screen is 

15 marketed under the trade name "HoloClear" and manufactured by Dai Nippon 
Printing; such material is translucent to light shined onto it at a 35 degree angle. 
This screen takes the place of the IR-transparent VIS-translucent screen or the 
scattering polarizer screen. If the projector shines light onto the screen from that 
angle, then there will be no glare from the projector. As long as the camera is at 

20 a significantly different angle (to the screen) from the projector, the system will 
be able to function properly. 
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Interface to Window Unit 

[01 73] The interface to the window unit can be made distance-dependent with 
the same methods as the self-contained unit, including the use of stereo 
cameras, time-of-flight cameras, and the techniques described in the 
5 "Touchscreen" sections of this patent. In one embodiment, the interactions with 
the window unit include a mix of full-body interactions and more precise 
gestures, such as pointing. 

[0174] With a vision system that extracts depth information, it is possible to 
10 isolate pointing gestures and other hand motions through several methods. 
First, the camera image can be divided up into distance ranges, with one 
distance range for full-body interaction and another (presumably closer) distance 
range for pointing gestures. The objects in the latter distance range would be 
tracked as pointers, with their locations serving as input to the application 
15 running on the display. Alternatively, all objects less than a particular distance 
from the screen could be analyzed to find the point within them that is closest to 
the screen. If that point is closer to the screen from the rest of the object by at 
least a particular threshold, then it could become an input for pointing gestures. 
Visual feedback on the screen could be provided to show the position of any 
20 detected hand motions. 

Techniques for Reducing Glare 

[01 75] If it can be ensured that the viewer always sees the projector glare from a 
specific range of angles, then the glare can be further reduced without adversely 
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affecting the camera's view of the scene. Figures 23A, 23B and 23C are 
simplified schematic diagrams respectively illustrating various techniques for 
reducing glare in accordance with different embodiments of the present 
invention. Figure 23A illustrates a pleated screen material. Suppose that the 
5 screen is pleated so that a light ray coming from an oblique angle would have to 
go through several layers of screen, while a light ray coming close to straight 
through would usually only go through one layer. If the projector's light comes 
from an oblique angle while the camera views the screen closer to parallel, then 
the amount of scattering of the projector's light would be greatly increased 
10 without adversely affecting the camera's view. In Figure 23A, the majority of the 
camera's view only goes through one screen layer, while all of the projected light 
goes through multiple layers. 

[0176] There are several ways.of achieving the desired effect. Instead of a 
15 pleated screen, a flat screen could be supplanted with a microlouver material to 
create the same effect as a pleated screen, as shown in Figure 23B. 
Alternatively, small, flat sheetlike particles of screen material could be added (all 
oriented horizontally) to a transparent substrate, as shown in Figure 23C. In all 
cases, a typical light ray approaching the screen from an oblique angle 
20 encounters far more scattering material than a typical light ray that is 
perpendicular to the material. 

[0177] The texture of the screen in all cases should be small enough that the 
viewer would not notice it. This technique is most useful when using an off-axis 
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projector; however, it is useful in any situation where the projector and camera 
are viewing the screen from different angles. 

[0178] It is important to prevent the infrared light source from shining on the 
5 screen and reducing the contrast. Thus, if the infrared illuminator is placed 

behind the screen, it is advantageous to place the infrared illuminator at an angle 
for which the screen's scattering is minimized. 

[0179] Alternatively, view control film products (such as Lumisty), which are 
10 translucent at a narrow range of viewing angles and transparent at all other 
angles, can help reduce glare in some cases. 

[0180] By placing the projector at a particular angle to the screen, it can be 
ensured that anyone looking directly into the projector's beam will be looking at 

1 5 the screen at an angle for which the view control film is translucent. Figures 24A 
and 24B show one method by which view control film can reduce glare. Figure 
24A shows the experience of a person (or camera) viewing light through one kind 
of view control film. Light coming from the translucent region is diffused, 
reducing or eliminating any glare from light sources in this region. Light sources 

20 from the two transparent regions will not be diffused, allowing the person or 
camera to see objects in these regions. The boundaries of the region of view 
control film are often defined by a range of values for the angle between a light 
ray and the view control film's surface along one dimension. Figure 24B shows 
the view control film in a sample configuration for reducing glare on an interactive 
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window display. The view control film is used in conjunction with the IR- 
transparent VIS-translucent screen. Because of the angle of the projector, it is 
impossible to look directly into the projector's beam without being at an angle at 
which the view control film diffuses the light. However, the camera is able to 
5 view objects through the view control film because the camera is pointed at an 
angle at which the view control film is transparent. Thus, glare is reduced 
without affecting the camera's ability to view the scene. 

Exemplary Configuration 

[0181] One exemplary embodiment of a window-based display 2500 utilizes a 
scattering polarizer as a screen, as shown in Figure 25. This embodiment is an 
interactive window display 2500 in which all the sensing and display components 
necessary to make the display work are placed behind the window 2505. The 
window display 2500 allows users in front of the window 2505 to interact with 
video images displayed on the window 2505 through natural body motion. 

[0182] In one embodiment, the displayed image is generated by an LCD 
projector 2510. In most LCD projectors, red and blue light are polarized in one 
direction, while green is polarized in a perpendicular direction. A color selective 
20 polarization rotator 2515, such as the retarder stack "ColorSelect" technology 

produced by the ColorLink Corporation, is used to rotate the polarization of green 
light by 90 degrees. Alternatively, the polarization of red and blue light can be 
rotated by 90 degrees to achieve the same effect. By placing color selective 
polarization rotator 2515 in front of projector 2510, all the projector's light is in the 
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same polarization. The scattering polarizer 2525 is oriented so that the direction 
of maximum scattering is parallel to the polarization of the projector's light. Thus, 
when this projector's light reaches scattering polarizer 2525, it is all scattered, 
providing an image for the user on the other side of the screen. 

5 

[0183] A video camera 2530 sensitive to only near-infrared light views the area 
in front of the screen, referred to as the "camera's field of view". Objects within 
this field of view will be visible to camera 2530. Illumination for the camera's field 
of view comes from sets of infrared LED clusters 2535, which produce light in 
10 wavelengths viewable by camera 2530, on the back side of the screen. Note 
that the camera's field of view is slanted upward so that only people who are 
near the screen fall within the field of view. This prevents objects distant from 
the screen from affecting the interactive application that uses camera 2530 as 
input. 

15 

[0184] The paths of visible and infrared light through the exemplary embodiment 
in Figure 25 will now be described. The two perpendicular polarizations of light 
are referred to as polarization A and polarization B. 



20 [0185] Visible light emerges from LCD projector 2510, with red and blue light in 
polarization A and green in polarization B. This light first passes through color 
selective polarization rotator 2515, which leaves the red and blue light 
unaffected, but rotates the polarization of green light such that it is in polarization 
A. Next, this light passes through a linear polarizer 2520, which transmits light in 
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polarization A and absorbs light in polarization B. This linear polarizer 2520 
"cleans up" the light - it absorbs any of the projector's light which is still B- 
polarized. Next, the light passes through scattering polarizer 2525, which is 
oriented to scatter light in polarization A and transmit light in polarization B. 
5 Thus, nearly all of the projector's light is scattered. Note that this scattered light 
retains its polarization A. Optionally, the light may then pass through a linear 
polarizer 2540 which transmits light in polarization A and absorbs light in 
polarization B. This polarizer tends to improve image quality. 

10 [0186] The infrared light emitted from infrared illuminators 2535 may begin 

unpolarized. Optionally, for improved clarity, this light can first pass through an 
infrared linear polarizer to polarize it in polarization B so that less of it will be 
scattered by scattering polarizer 2525. If the light is unpolarized, some of it will 
be scattered as it passes through scattering polarizer 2525, but the light of 

15 polarization A will pass through scattering polarizer 2525 unscattered. Since the 
wavelength of the infrared light is sufficiently long, it passes unaffected through 
any visible-light linear polarizers 2540 and can illuminate objects in front of the 
screen, such as a human user. 



20 [01 87] Infrared light returning from in front of window 2505 toward camera 2530 
will be unaffected by linear polarizer 2540. However, as the light passes through 
scattering polarizer 2525, the light of polarization A will be scattered while the 
light of polarization B will remain unscattered. The camera 2530 has an infrared 
linear polarizer 2545 immediately in front of it; this polarizer 2545 absorbs light of 
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polarization A and transmits light of polarization B. Thus, camera 2530 only 
views light of polarization B, which was left unscattered by scattering polarizer 
2545. This gives camera 2530 a clear, high-contrast image of the area in front of 
the screen. 

5 

Using Prismatic Films 

[0188] In interactive projected window displays in general, it is often desirable to 
have the area under the window display clear, but position the camera so that 
the area it views slopes upward. This situation can be achieved with the use of 
10 prismatic films that redirect all light that pass through them by a specific angle. 
For examples, the Vikuiti IDF film made by 3M redirects incoming light by 20 
degrees. By placing one or more of these films on either side of the projection 
screen to redirect light upward, the camera can be placed higher relative to the 
screen, as shown in Figure 26. 

15 

Compaction 

[0189] The overall size of the system can be compacted using a mirror. Figure 
27 shows a configuration in which the camera and projector are placed next to 
the window, pointing away from it, and a mirror reflects their light beams back 
20 toward the window. 

Camera Improvements 

[0190] For further image quality improvement and ambient light rejection, the 
camera and camera's illuminators can be strobed together. This approach is 
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fully compatible with the use of various software and hardware approaches for 
3D imaging. In particular, the camera and illuminator in this design can be 
replaced with a time-of-flight camera. 

Visible Light System 

[0191] If no linear polarizers are added next to the scattering polarizer (as 
shown in Figure 25), then the design does not require the use of an infrared 
camera. A color or black-and-white visible light camera is able to image the area 
in front of the screen, so long as there is a visible-light linear polarizer 
immediately in front of the camera, with the polarization direction parallel to the 
direction for which the scattering polarizer is transparent. Thus, the projected 
image is unseen by the camera, allowing the camera to see the area in front of 
the screen unimpeded. This camera can work either with the existing ambient 
light or with additional visible lights placed near the display to illuminate users 
and objects in front of the screen. If additional visible lights are added, the 
camera and lights may be strobed together, as described in the section of the 
present application entitled "Directional Ambient Infrared" to improve the quality 
of the camera^ images and limit the effects of ambient and projector light on the 
image. 

[0192] For further image quality improvement, a high-speed aperture can be 
placed in front of the projectors lens. This aperture may be mechanical or 
electronic; one available electronic aperture is the liquid-crystal-based Velocity 
Shutter, produced by Meadowlark Optics. In one embodiment, this shutter 
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remains open nearly all of the time, allowing the projector light to pass through. 
The shutter only closes to block the projector's light while the camera is taking a 
picture. If the camera's exposure time is brief, then the brightness of the 
projector will be almost unaffected. Note that the use of a velocity shutter to 
5 block the projector's light during the camera's exposure also allows a visible-light 
camera to be used in front projected interactive floor or wall displays. 

[0193] Note that with the use of a visible-light camera in an interactive video 
projection system, a real-time picture of the people in front of the screen (the 

10 system's users) is obtained in addition to a vision signal classifying each pixel of 
the camera's image as foreground or background. This allows the vision system 
to isolate a picture of the system's users with the static background removed. 
This information allows the system to place a color image of the users in the 
image it displays, with artificially generated images inserted on and around the 

15 users. If this system is properly calibrated, a user could touch the screen, and 
the displayed image of the user would touch the same location on the screen at 
the same time. These features provide significant improvement in the quality of 
interactive applications running on this display including, for example, allowing 
users to literally see themselves placed inside the interactive content. 

20 

[0194] The visible-light image from the camera can be used to create a virtual 
mirror, which looks and behaves like a real mirror, but the mirror image can be 
electronically manipulated. For example, the image could be flipped horizontally 
to create a non-reversing mirror, in which users see an image of themselves as 
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other people see them. Alternatively, the image could be time-delayed so that 
people could turn around to see their own backs. This system could thus have 
applications in environments where mirrors are used including, for example, 
dressing rooms. 

5 

TlME-OF-FLlGHT CAMERA INTERACTIVE DISPLAY 

[01 95] Embodiments of the present invention may be implemented using time- 
of-flight cameras. A time-of-flight camera has a built-in capability to detect 
distance information for each pixel of its image. Using a time-of-flight camera 

10 eliminates the need for a modified display. In other words, the time-of-flight 
camera may work with any display (e.g., an LCD panel, a cathode-ray tube 
display, etc.) without modifications. A single time-of-flight camera may be used. 
However, a single time-of-flight camera may not be able to detect objects that 
are blocked by objects closer to the camera. Therefore, embodiments of the 

15 present invention utilize multiple time-of-flight cameras, as shown in Figure 28. 

[0196] With redundancy of cameras, there is no longer a need to worry about 
one camera not being able to detect al the objects because of one object 
occluding another object. For example, as shown in Figure 29, four time-of-flight 
20 cameras may be placed at the corners of a display, ensuring that the entire area 
of the display is interactive. In order to use this time-of-flight implementation for 
multiple cameras, a coordinated transform is performed on each pixel of each 
time-of-flight camera to put it in a common coordinate space. One such space is 
defined by: (x, y) - the position of the point projected perpendicular onto the 
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display surface, and (z) - the distance from the display. This coordinate space 
transformation can be determined by looking at the angle of each camera (and 
position) relative to the screen. Alternatively, the transformation may be 
determined by a calibration process, in which an object of known size, shape and 
5 position is placed in front of the screen. By having each of the cameras image 
the object, the appropriate transformation function can be determined from 
points viewed by each camera into points in the common coordinate space. If 
the camera coordinate transforms are done in real time, then a real-time picture 
of the area in front of the camera in 3D is achieved. 

10 

Uses 

[0197] The interactive video display system can be used in many different 
applications. The system's capability to have touchscreen-like behavior as well 
as full or partial body outline interaction increases its appeal for information 
15 interfaces which require more precise selection and manipulation of buttons and 
objects. 

[0198] Uses of the transparent-display-screen-based and projector-based 
interactive display systems include, but are not limited to, interactive video 
20 games in which users move their bodies to play the game, interactive menu, 
catalog, and browsing systems that let users browse through pages of 
informational content using gestures, systems that allow users to "try on" clothing 
using an image of themselves, pure entertainment applications in which images 
or outlines of users serve as input to a video effects system, interactive 
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characters that interact with the motions of users in front of the screen, and 
virtual playlands and storybooks that users interact with by moving their bodies. 

[0199] Other uses of the present invention include, but are not limited to: 
5 allowing users to customize or view available options for customizing the product 
on display, allowing the product on display to be ordered at the display, using 
either the display interface, a keyboard, a credit card swiper, or a combination of 
the three, comparing the features of multiple products on the display, showing 
combinations or compatibilities between multiple products on the display, and 
10 placing a product in different virtual settings on the screen to demonstrate the 
features (e.g., water, forest, asphalt, etc.) 

Peripherals 

[0200] Transparent-display-screen-based and projector-based interactive 
15 display systems can incorporate additional inputs and outputs, including, but not 
limited to, microphones, touchscreens, keyboards, mice, radio frequency 
identification (RFID) tags, pressure pads, cellular telephone signals, personal 
digital assistants (PDAs), and speakers. 

20 [0201] Transparent-display-screen-based and projector-based interactive 
display systems can be tiled together to create a single larger screen or 
interactive area. Tiled or physically separate screens can also be networked 
together, allowing actions on one screen to affect the image on another screen. 
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[0202] In an exemplary implementation, the present invention is implemented 
using a combination of hardware and software in the form of control logic, in 
either an integrated or a modular manner. Based on the disclosure and 
teachings provided herein, a person of ordinary skill in the art will know of other 
ways and/or methods to implement the present invention. 

[0203] In broad summary, this writing has disclosed a self-contained interactive 
video display system. A flat-panel display screen displays a visual image for 
presentation to a user on a front side of the flat-panel display screen. A first 
illuminator illuminates the flat-panel display screen with visible light. A second 
illuminator illuminates an object. A camera detects interaction of an illuminated 
object with the visual image, wherein the camera is operable to view the object 
through the flat-panel display screen. A computer system directs the projector to 
change the visual image in response to the interaction. 

[0204] In one exemplary aspect, the present invention as described above 
provides a system that allows a camera to view an area in front of a display. In a 
related invention, a system is provided to create a reactive space in front of the 
display. The present invention can be used to capture information from the 
reactive space. 

[0205] It is understood that the examples and embodiments described herein 
are for illustrative purposes only and that various modifications or changes in 
light thereof will be suggested to persons skilled in the art and are to be included 
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within the spirit and purview of this application and scope of the appended 
claims. All publications, patents, and patent applications cited herein are hereby 
incorporated by reference for all purposes in their entirety. 

5 [0206] Various embodiments of the invention, a self-contained interactive video 
display system, are thus described. While the present invention has been 
described in particular embodiments, it should be appreciated that the invention 
should not be construed as limited by such embodiments, but rather construed 
according to the below claims. 
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CLAIMS 

What is claimed is: 

1 . A self-contained interactive video display system comprising: 
5 a display screen for displaying a visual image for presentation to a user on 

a front side of said flat-p&nel display screen; 

a first illuminator for illuminating said display screen with visible light; 
a second illuminator for illuminating an object; 

a camera for detecting interaction of an illuminated object with said visual 
10 image, wherein said camera is operable to view said object through said screen; 
and 

a computer system for directing said display screen to change said visual 
image in response to said interaction. 

15 2. The system of Claim 1 wherein the display screen is a flat-panel 

display^creen. 

3. The self-contained interactive video display system as recited in 
Claim 2 wherein said flat-panel display screen is a liquid crystal display (LCD) 

20 panel. 

4. The self-contained Interactive video display system as recited in 
Claim 1or*2 wher6in said second illuminator is an infrared illuminator for 
illuminating said object with infrared illumination, and wherein said camera is an 
infrared camera for detecting infrared illumination. 
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5. The self-contained interactive video display system as recited in 
Claim 1 or 2 further comprising a diffusing screen for diffusing light of said first 
illuminator. 

6. The self-contained interactive video display system as recited in 
Claim 1 or 2 wherein said camera, said first illuminator, said second illuminator, 

A. 

and said computer system are comprised within an enclosure, and wherein one 
side of said enclosure comprises said flat-panel display screen. 

7. The self-contained interactive video display system as recited in 
Claim 1 or 2, wherein said second illuminator projects illumination through said 
flat-panel display screen onto said object and wherein said second illuminator is 
positioned so as to reduce potential for glare effects on said camera. 

8. The self-contained interactive video display system as recited in 
Claim 1 or 2 wherein said second illuminator is located next to said flat-panel 
display screen such that said second illuminator does not project illumination 
through said flat-panel display screen. 

9. The self-contained interactive video display system as recited in 
Claim 1 or 2 comprising a plurality of said second illuminators, wherein said 
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illuminators are located next to said flat-panel display screen and do not project 
illumination through said flat-panel display screen and are also located behind 
said screen. 

10. The self-contained interactive video display system as recited in 
Claim 1 or 2 wherein said camera is located behind said flat-panel display screen 
and pointed toward said screen, allowing said camera to view the area on and in 
front of the screen. 

1 1 . The self-contained interactive video display system as recited in 
Claim 1 or 2 further comprising a series of mirror strips positioned at a distance 
from said screen to correct distortion of a view of said camera. 

12. The self-contained interactive video display system as recited in 
Claim 1 or 2 further comprising a Fresnel lens positioned adjacent to said screen 
to correct distortion of a view of said camera. 

13. The self-contained interactive video display system as recited in 
Claim 1 or 2 further comprising a wavelength-based diffuser positioned adjacent 
to said flat-panel display screen. 

14. The self-contained interactive video display system as recited in 
Claim 13, where said wavelength-based diffuser is substantially transparent to 

infrared light and substantially translucent to visible light. 
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15. The self-contained interactive video display system as recited in 
Claim 13, where said wavelength-based diffuser is a material that causes 
Rayleigh scattering. 

5 

16. The self-contained interactive video display system as recited in 
Claim 1 or 2 further comprising a diffuser positioned adjacent to said flat-panel 
display screen, wherein said diffuser is substantially translucent to light passing 
through said diffuser at an oblique angle and substantially transparent to light 

10 passing through said diffuser at a substantially perpendicular angle, wherein said 
first illuminator is placed at an oblique angle to said diffuser. 

17. The self-contained interactive video display system as recited in 
Claim 1 or 2 further comprising a scattering polarizer positioned adjacent to said 

1 5 flat-panel display screen and for scattering light from said first illuminator, and 
wherein said camera is sensitive to light of a polarization not scattered by said 
scattering polarizer. 

18. The self-contained interactive video display system as recited in 
20 Claim 17, wherein said flat-panel display is a liquid crystal display panel, and 

wherein said scattering polarizer is oriented such that light polarized in a 
direction for which said scattering polarizer scatters light passes through said 
liquid crystal display panel and light polarized in a direction for which said 
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scattering polarizer does not scatter light is absorbed by said liquid crystal 
display panel. 

19. The self-contained interactive video display system as recited in 

5 Claim 17, further comprising a linear polarizer for polarizing light received at said 
camera at wavelengths to which said camera is sensitive, so as to allow said 
camera to ignore light scattered by said scattering polarizer. 

20. The self-contained interactive video display system as recited in 
10 Claim 1 or 2 further comprising a diffusing material that can change from 

substantially translucent to substantially transparent, wherein said diffusing 
material is substantially translucent when said first illuminator is visible to a 
human user on said front side of said flat-panel display screen, and is 
substantially transparent when said camera is detecting objects in front of said 
15 flat-panel display screen, wherein said diffusing material is placed behind said 
flat-panel display screen. 

21 . The self-contained interactive video display system as recited in 
Claim 1 or 2, wherein said second illuminator is strobed in time with exposures of 
said camera. 

20 

22. The self-contained interactive video display system as recited in 
Claim 1 or 2 wherein an image of said camera is calibrated to said visual image 
such that said interaction caused by said object is matched to a physical position 

of said object proximate to said the screen. 

89 



WO 2005/057399 



PCT/US2004/041320 



23. The self-contained interactive video display system as recited in 
Claim 1 or 2 wherein said self-contained interactive video display system is 
operable to determine information about the distance of said object from the said 
screen. 

24. The self-contained interactive video display system as recited in 
Claim 23 wherein said camera is a stereo camera. 

25. The self-contained interactive video display system as recited in 
Claim 23 wherein said camera is a time-of-flight camera. 

26. The self-contained interactive video display system as recited in 
Claim 25 wherein said time-of-flight camera is positioned such that said time-of- 
flight camera does not reflect back onto itself. 

27. The self-contained interactive video display system as recited in 
Claim 1 or 2 wherein said self-contained interactive video display system 
provides touchscreen functionality when said object is proximate said screen. 

28. The self-contained interactive video display system as recited in 
Claim 27 further comprising a transparent touchscreen adjacent said front side of 
said screen. 
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29. The self-contained interactive video display system as recited in 
Claim 27 further comprising an edge-lit transparent sheet adjacent said front side 
of said screen, and wherein said camera is operable to distinguish light created 
when said object comes in contact with said edge-lit transparent sheet. 

5 

30. The self-contained interactive video display system as recited in 
Claim 1 or 2 wherein said object is a body part of a human user. 

31 . The self-contained interactive video display system as recited in 
10 Claim 1 or 2, wherein said camera and said second illuminator comprise a time- 
of-flight camera. 

32. The self-contained interactive video display system as recited in 
Claim 1 or 2, wherein a plurality of time-of-flight cameras are placed in a manner 

15 so as to provide complete coverage of the area in front of the display and angled 
so as to prevent specular reflection into said time-of-flight cameras due to said 
screen. 



33. A method for presenting an interactive visual image using a self- 
20 contained interactive video display system, said method comprising: 

displaying a visual image on a flat-panel display screen, wherein said 
visual image is for presentation to a user on a front side of said flat-panel display 
screen; 

illuminating a back side of said flat-panel display screen with visible light; 
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illuminating an object on said front side of said flat-panel display screen; 
detecting interaction of said object with said visual image through said flat- 
panel display screen; and 

changing said visual image in response to said interaction. 

5 

34. The method as recited in Claim 33 wherein illuminating said object 
comprises illuminating said object with infrared illumination, and wherein said 
detecting said interaction comprises detecting infrared illumination projected onto 
said object. 

10 

35. The method as recited in Claim 33 wherein said self-contained 
interactive video display system is comprised within an enclosure, and wherein 
one side of said enclosure comprises said flat-panel display screen. 

15 36. The method as recited in Claim 33, wherein said illuminating said 

object comprises projecting illumination through said flat-panel display screen 
onto said object. 

37. The method as recited in Claim 33 further comprising correcting 
20 distortion using a Fresnel lens positioned adjacent to said screen. 

38. The method as recited in Claim 33 further comprising an infrared- 
transparent visible light-translucent screen positioned adjacent to said flat-panel 
display screen. 
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39. The method as recited in Claim 33 further comprising a transparent 
screen having a physical texture positioned adjacent to said flat-panel display 
screen and for scattering light from said illuminating said back side of said flat- 
panel display screen with visible light. 

40. The method as recited in Claim 33 further comprising a transparent 
screen of Rayleigh scattering material positioned adjacent to said flat-panel 
display screen and for scattering light from said illuminating said back side of 
said flat-panel display screen with visible light. 

41 . The method as recited in Claim 33 further comprising a scattering 
polarizer positioned adjacent to said flat-panel display screen and for scattering 
light from said illuminating said back side of said flat-panel display screen with 
visible light. 

42. The method as recited in Claim 33 wherein said self-contained 
interactive video display system further comprises a wavelength-based diffuser 
positioned adjacent to said flat-panel display screen. 

43. The method as recited in Claim 33 wherein said self-contained 
interactive video display system further comprises a diffusing material that can 
change from substantially translucent to substantially transparent, wherein said 
diffusing material is substantially translucent during said illuminating said back 
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side of said flat-panel display screen with visible light, and is substantially 
transparent during said detecting interaction of said object with said visual image 
through said flat-panel display screen, wherein said diffusing material is placed 
behind said flat-panel display screen. 

44. The method as recited in Claim 33, wherein said illuminating said 
object is strobed in time with said detecting. 

45. The method as recited in Claim 33 further comprising calibrating an 
image captured at said detecting to said visual image such that said interaction 
caused by said object is matched to a physical position of said object proximate 
to said screen. 

46. The method as recited in Claim 33 further comprising determining 
information about a distance of said object from said screen. 

47. The method as recited in Claim 46 wherein said detecting is 
performed by a stereo camera. 

48. The method as recited in Claim 46 wherein detecting is performed 
by a time-of-flight camera. 
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49. The method as recited in Claim 48 further comprising positioning 
said time-of-flight camera such that the time-of-flight camera does not reflect 
back onto itself. 

50. The method as recited in Claim 33 wherein further comprising 
providing touchscreen functionality when said object is proximate said screen. 

51 . The method as recited in Claim 50 further comprising providing a 
transparent touchscreen adjacent said front side of said screen. 

52. The method as recited in Claim 50 further comprising providing an 
edge-lit transparent sheet adjacent said front side of said screen, and wherein 
said detecting is operable to distinguish light created when said object comes in 
contact with said edge-lit transparent sheet. 

53. The method as recited in Claim 33 wherein said object is a body 
part of a human user. 

54. The system of Claim 1 wherein said display screen is: 

a liquid crystal display screen, said first illuminator is a visible light 
illuminator, said second illuminator is an infrared illuminator, and said second 
camera is an infrared camera 
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wherein said infrared camera, said visible light illuminator, said infrared 
illuminator, and said computer system are comprised within an enclosure, and 
wherein one side of said enclosure comprises said liquid crystal display screen. 

55. The self-contained interactive video display system as recited in 
Claim 54 further comprising a diffusing screen for diffusing light of said visible 
light illuminator. 

56. The self-contained interactive video display system as recited in 
Claim 54, wherein said infrared illuminator projects illumination through said 
liquid crystal display screen onto said object. 

57. The self-contained interactive video display system as recited in 
Claim 56, wherein said infrared illuminator is positioned so as to reduce potential 
for glare effects pn said infrared camera. 

58. The self-contained interactive video display system as recited in 
Claim 54 wherein said infrared illuminator is located next to said liquid crystal 
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display screen such that said infrared illuminator does not project illumination 
through said liquid crystal display screen. 

59. The self-contained interactive video display system as recited in 
5 Claim 54 comprising a plurality of said infrared illuminators, wherein said infrared 
illuminators are located next to said flat-panel display screen and do not project 
illumination through said flat-panel display screen and are also located behind 
said screen. 

10 60. The self-contained interactive video display system as recited in 

Claim 54 wherein said infrared camera is located behind said liquid crystal 
display screen and pointed toward said screen, allowing said infrared camera to 
view the area on and in front of the screen. 

15 61 . The self-contained interactive video display system as recited in 

Claim 54 further comprising a series of mirror strips positioned at a distance from 
said screen to correct distortion of a view of said infrared camera. 

62. The self-contained interactive video display system as recited in 
20 Claim 54 further comprising a Fresnel lens positioned adjacent to said screen to 
correct distortion of a view of said infrared camera. 
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63. The self-contained interactive video display system as recited in 
Claim 54 further comprising a wavelength-based diffuser positioned adjacent to 
said liquid crystal display screen. 

64. The self-contained interactive video display system as recited in 
Claim 63, where said wavelength-based diffuser is substantially transparent to 
infrared light and substantially translucent to visible light. 

65. The self-contained interactive video display system as recited in 
Claim 63, where said wavelength-based diffuser is a material that causes 
Rayleigh scattering. 

66. The self-contained interactive video display system as recited in 
Claim 54 further comprising a diffuser positioned adjacent to said liquid crystal 
display screen, wherein said diffuser is substantially translucent to light passing 
through said diffuser at an oblique angle and substantially transparent to light 
passing through said diffuser at a substantially perpendicular angle, wherein said 
visible light illuminator is placed at an oblique angle to said diffuser. 

67. The self-contained interactive video display system as recited in 
Claim 54 further comprising a scattering polarizer positioned adjacent to said 
liquid crystal display screen and for scattering light from said visible light 
illuminator, and wherein said infrared camera is sensitive to light of a polarization 
not scattered by said scattering polarizer. 
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68. The self-contained interactive video display system as recited in 
Claim 67, wherein said liquid crystal display is an liquid crystal display panel, and 
wherein said scattering polarizer is oriented such that light polarized in a 

5 direction for which said scattering polarizer scatters light passes through said 
liquid crystal display panel and light polarized in a direction for which said 
scattering polarizer does not scatter light is absorbed by said scattering polarizer. 

69. The self-contained interactive video display system as recited in 
10 Claim 67, further comprising a linear polarizer for polarizing light received at said 

infrared camera at wavelengths to which said infrared camera is sensitive, so as 
to allow said infrared camera to ignore light scattered by said scattering polarizer. 

70. The self-contained interactive video display system as recited in 
15 Claim 54 further comprising a diffusing material that can change from 

substantially translucent to substantially transparent, wherein said diffusing 
material is substantially translucent when said first illuminator is visible to a 
human user on said front side of said flat-panel display screen, and is 
substantially transparent when said infrared camera is detecting objects in front 
20 of said liquid crystal display screen, wherein said diffusing material is placed 
behind said liquid crystal display screen. 
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71 . The self-contained interactive video display system as recited in 
Claim 54, wherein said infrared illuminator is strobed in time with exposures of 
said infrared camera. 

72. The self-contained interactive video display system as recited in 
Claim 54 wherein an image of said infrared camera's image is calibrated to said 
visual image such that said interaction caused by said object is matched to a 
physical position of said object proximate to said the screen. 

73. The self-contained interactive video display system as recited in 
Claim 54 wherein said self-contained interactive video display system is operable 
to determine information about the distance of said object from the said screen. 

74. The self-contained interactive video display system as recited in 
Claim 73 wherein said infrared camera is a stereo infrared camera. 

75. The self-contained interactive video display system as recited in 
Claim 73 wherein said infrared camera is a time-of-flight infrared camera. 

76. The self-contained interactive video display system as recited in 
Claim 75 wherein said time-of-flight infrared camera is positioned such that said 
time-of-flight infrared camera does not reflect back onto itself. 
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77. The self-contained interactive video display system as recited in 
Claim 54 wherein said self-contained interactive video display system provides 
touchscreen behavior functionality when said object is proximate said screen. 

5 78. The self-contained interactive video display system as recited in 

Claim 77 further comprising a transparent touchscreen adjacent said front side of 
said screen. 

79. The self-contained interactive video display system as recited in 

10 Claim 77 further comprising an edge-lit transparent sheet adjacent said front side 
of said screen, and wherein said infrared camera is operable to distinguish light 
created when said object comes in contact with said edge-lit transparent sheet. 

80. The self-contained interactive video display system as recited in 
1 5 Claim 54 wherein said object is a body part of a human user. 

81. The self-contained interactive video display system as recited in 
Claim 54, wherein said infrared camera and said infrared illuminator comprise a 
time-of-flight infrared camera. 

20 

82. The self-contained interactive video display system as recited in 
Claim 81 , wherein a plurality of time-of-flight infrared cameras are placed in a 
manner so as to provide complete coverage of the area in front of the display 
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and angled so as to prevent specular reflection into said time-of-flight infrared 
cameras due to said screen. 
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